Virtualization Technology News and Information
SWsoft Looks at VMmark and SPEC Benchmarking

Ilya Baimetov, Director of Technology at SWsoft, posted on the SWsoft SaaS Blog, his comparison of VMmark and the SPEC CPU benchmark.  He writes:

  • Workload mix – For some reason, VMware uses their Enterprise software (ESX) to run what seems to be an SMB scenario (mix of different workloads on the same machine). Typically, in the enterprise there is a special server configuration (CPU/RAM/Disk/Network) for each workload – web server, Exchange server, low-end DB server, high-end DB server and so on. It'd be only logical to consolidate similar workloads on the similarly configured servers – to maximize resource usage.
  • Workload scores. SPEC CPU runs all the workloads sequentially, measuring the maximum performance of each workload on the machine, normalizing against the reference system and then averaging across the workloads. You get an aggregate score, but you can also find out how well the machine is suited for running a certain workload that is of the most interest for you. With VMmark, it's always a wild mix. VMmark is useless if you want to find out how well a server is suited for consolidating Exchange or SQL Server.
  • Aggregate score. Because it's always a mix of workloads, each workload in a tile is throttled and never runs at the full speed. And, if the server is so powerful a single tile cannot load it fully, you are supposed to add another, which would double the number of VMs and probably skew the results. Bottom line, I'm not sure what exactly VMmark measures.
  • Underlying platform – SPEC CPU runs on any hardware and any OS. The VMmark cannot be run on Virtuozzo because it executes Windows and Linux workloads in parallel on the same machine.

Ilya thinks that a better approach would be to use the SPEC CPU methodology – run multiple workloads serially and average out the results. Specifically:

  • Measure maximum aggregate performance of 1, 2, 4, 8 and 16 virtual environments with the same workload on the same machine. Normalize and average.
  • Repeat that for each workload and then average the results.

The benefits of this approach:

  • You can easily see what the virtualization overhead is for each workload depending on the number of concurrent VMs.
  • Virtuozzo will be able to run the benchmark because only one OS is used for a single benchmark run.

What do you think?  Do you have any experience running these two benchmarking tools? 

When I first downloaded VMmark, I was a little surprised that it required SPEC tools.  With all the free benchmarking tools, and all the money and energy that VMware could have put behind creating a true virtualization benchmarking tool, I was a little disappointed that I would have to spend $1700 on two benchmarking tools to completely use VMmark.

** Update **

 The original article is going to be posted on the Virtuozzo Blog.  Stay tuned and watch for it.  It was originally posted on their SaaS blog, but is going to be moved.  Sorry for the confusion.

Published Friday, August 10, 2007 5:13 AM by David Marshall
Filed under:
massimo - (Author's Link) - August 11, 2007 9:22 AM


I can't find the article any more ... the link seems to be broken. However:


>It'd be only logical to consolidate

>similar workloads on the similarly

>configured servers – to maximize

>resource usage.


I don't understand this. In my opinion this is exactly the opposite (i.e. you would want to run different workloads on the same system to maximise hw resources). If you have a 100 x CPU intensive workloads on your high-end server and those workloads are doing little in terms of memory / network and disk that is not optimizing your workload.... (in my opinion). Their negative comments on VMark seem to be biased by their inability to use it (because of their OS virtualization architecture) rather than by common sense. VMark is not perfect but certainly the idea is strong and its basic concept (perhaps not the name) is here to stay.

As per :

>With all the free benchmarking tools,

>and all the money and energy that

>VMware could have put behind

>creating a true virtualization

>benchmarking tool, I was a little >disappointed that I would have to

>spend $1700 on two benchmarking

>tools to completely use VMmark

See my comments here:


David Marshall - August 11, 2007 10:07 AM

Hi Massimo, I agree, simple benchmarking CAN lead to nowhere.  I guess my biggest complaint with VMmark is that it isn't a self-contained virtualization benchmarking package.  And if you offer a free benchmarking solution, why add in a required (and expensive) add-in benchmarking package.  Why not (I can't believe I'm saying this) re-invent the wheel on this one?  

Still in agreement with you on the whole huge lab and time to implement requirement.  Just a hard pill to swallow calling it a free benchmark tool only to find out you need to spend $1700 for some other tool to fully implement the "free" one.  :)

massimo - (Author's Link) - August 11, 2007 5:18 PM

Ah ok I see what u are saying ....

I don't think that the problem lies too much in the 1700$ you need to pay for the tools royalties that VMmark uses but rather in the fact that EVEN THOUGH VMware was to provide a full free benchmarking package... the infrastructure and the associated time required to run it would be enormous and would off-set the 1700$ license fees. It would be like trying to run a TPC-C benchmark in house ..... if you are not HP / IBM / DELL / SUN it's nearly impossible.

I guess that the idea behind VMMark (or similar future virtualization benchmarks) is to create a standard methodology for server vendor and virtualization vendors to benchmark their systems / software rather than a free tool to give to end-users for playing around.


VMmark a server vendor leapfrog game - Server Farming - (Author's Link) - August 22, 2008 11:56 AM
To post a comment, you must be a registered user. Registration is free and easy! Sign up now!
<August 2007>