The evaluation of machine virtualization tools is a difficult exercise. We first motivated the use of microbenchmark to better understand the scalability limitations and merits of virtualization tools. We have described a set of metrics (overhead, linearity and isolation), and related microbenchmarks for the CPU, memory, disk and network resources. These metrics allow testing many aspects of these systems, performance as well as usability. We have compared 4 virtualization tools using this methodology: VMware, UML, Vserver and Xen. We clearly noticed strong limitations with VMware and UML, as previously published by other authors, but we have provided a detailed evaluation, identifying overhead, linearity and performance isolation limitations for all machine resources.  Vserver and Xen clearly provide the better performance. However, there is still room for improvements in Vserver and Xen, since they do not provide performance isolation for the network between VMs (which is desirable for some users) and Xen suffers from low intervirtual machine communication performance. A significant limitation of Vserver is that it cannot run kernels for guest virtual machines different to the hosting one. But, compared to Xen, its architecture saves a lot of memory space when running many virtual machines, since only one kernel is shared by all VMs. VMware has the advantage of providing performance isolation for all resources. It also allows running unmodified guest OS at the cost of a high overhead and poor linearity with respect to scalability.

According to their current respective merits and limitations, the compared VM technologies will match efficiently different application scenarios depending on their need in guest OS configuration, performance isolation and scalability. VMware clearly fits scenarios requiring small number of VMs and performance isolation between VMs. Such scenarios are likely to occur for Grid servers ensuring QoS from service level agreement (SLA). In addition, VMware accepts dynamic instantiation of user defined runtime environment, including specialized OS. Xen will match scenarios where many users or applications are deployed, possibly on ported OSes, on the same hardware with a ”best effort” or opportunistic like QoS (the performance of every VM will depend on the workload of the others). Vserver will accommodate more VMs and provide high performance communication between the VMs. But application should be compliant with the VM hosting kernel. Vserver will match for example scenarios where the number of physical nodes running a distributed application with a fixed number of processes may evolve from time to time. UML is the only one which can be runned by an unprivileged user.

Altogether we believe that 1) the result of this study will help users to select the VM technologies corresponding to the characteristics of their application and 2) the proposed metrics and benchmarks could help the VM designers  by evaluating their technology with a third point of view (close to user needs), between real applications and low level VM mechanisms benchmarks.

Read the entire paper, here.