Virtualization Technology News and Information
On Benchmarking Virtual Infrastructure

VMware's Engineering Performance Group recently posted on VMTN's "The Console" page.  In the discussion, they go into great detail to discuss the challenges of creating good benchmarking and performance metrics of a virtualized system.

In it, they write:

As virtualization becomes commonplace in the industry there is increasing interest in measuring the performance of virtualized platforms. Plenty of benchmarks exist to measure the performance of physical systems, but they fail to capture essential aspects of virtual infrastructure performance. We need a common workload and methodology for virtualized systems so that benchmark results can be compared across different platforms.

There are a number of unique challenges in creating sound and meaningful benchmarks for virtualized systems:

  • Capture the key performance characteristics of virtual systems
  • Ensure that the benchmark is representative of end user environments
  • Make the benchmark specification platform neutral
  • Define a single, easy to understand metric
  • Provide a methodical way to measure scalability so that the same benchmark can be used for small servers as well as larger servers

Let's take a look at these in turn.

Capture the key performance characteristics of virtual systems.

Users compare platforms based on their specific needs -- for example a user running a web server will compare the number of web requests that can be served by each platform, while a DBA will be more interested in the number of database transactions or simultaneous database connections. In the non-virtualized world users typically bind a single application to a single machine, and benchmarks have been developed to provide metrics for important application categories. For my examples of web servers and databases users could look at results for the industry standard SPECweb2005 and TPC-C benchmarks, respectively. Such benchmarks run the application to a point where some resource (usually CPU) is saturated and record the performance metric while respecting quality of service measures such as response time. Users not only compare platforms using the metrics provided by these benchmarks, but over time they build up expertise allowing them to relate their particular environment to published benchmark scores.

However, in a virtual environment, the typical usage of a machine is different from what is common on physical machines. One of the key benefits of virtualization is the ability to run multiple virtual machines on the same physical machine to increase the utilization of server resources. Multiple virtual machines running different operating systems and different applications with diverse resource requirements can all be running on the same machine. Moreover, these applications are typically not bottlenecked on any one resource, and have different (often conflicting) response time requirements. Running single application benchmarks one at a time and then aggregating their metrics might appear to be an easy solution, but that approach doesn’t work. Overall performance can be negatively impacted by competing resource demands among the workloads or positively impacted by optimizations such as transparent page sharing. A good virtualization benchmark must include multiple virtual machines running simultaneously.


Read the rest, here.

Published Thursday, September 14, 2006 1:35 PM by David Marshall
Filed under:
There are no comments for this post.
To post a comment, you must be a registered user. Registration is free and easy! Sign up now!
<September 2006>