Virtualization Technology News and Information
SPEC Releases New Cloud Benchmark

The Standard Performance Evaluation Corp. (SPEC) has released SPEC Cloud IaaS 2018, new benchmark software that measures the performance of infrastructure-as-a-service (IaaS) cloud implementations.

The SPEC Cloud IaaS 2018 benchmark measures replicated application instances, overall performance, relative scalability, and mean instance provisioning time of IaaS cloud implementations, both public and private. It replaces the 2016 version, which will be retired in three months.

Cloud providers, users of cloud services, vendors of cloud-related hardware and software, and researchers can use the benchmark to evaluate how different implementations affect cloud performance.

SPEC members participating in the cloud subcommittee include Dell, Digital Ocean, IBM, Intel, Oracle, Red Hat and VMware. Long-time SPEC benchmark developer Yun-seng Chao is a supporting contributor.

Flexible, representative workloads

Two multi-instance workloads are included within the SPEC Cloud IaaS framework:

-- The Yahoo! Cloud Serving Benchmark (YCSB) that uses the Cassandra NoSQL database to store and retrieve data in a manner representative of social media applications. -- Big data analytics based on a K-means clustering workload using Hadoop MapReduce.

The workloads are designed to test compute, storage and network performance of an IaaS cloud subject to strict quality of service (QoS) metrics. Each workload runs as a distributed application comprising six or seven cloud instances that make up an application instance. Adding new application instances increases the stress on the cloud infrastructure's instance-provisioning capabilities and on its available resources, including CPU, memory, disk and network.

The benchmark supports multi-tenancy and provides the flexibility to use a physical machine, a virtual machine, or a container as an instance type.

"SPEC Cloud IaaS 2018 incorporates improved methodologies that help ensure fairness, reliability and real-world rigor," says Paula Smith, chair of the SPEC Cloud subcommittee. "It provides metrics that go beyond relative performance and capacity measurement to show how typical cloud application workloads behave as underlying resources approach full capacity."

What's new for 2018

SPEC Cloud IaaS 2018 builds on the 2016 release with a variety of updates, including:

-- Easier installation of CBTOOL and workload images. -- Improved support for public clouds, OpenStack, and containers and container managers. -- Revised internal methodologies for scoring, reporting and metric calculations. -- New versions of Hadoop for K-means and Cassandra for YCSB. -- An increased I/O workload that tests the limits of current storage technologies.

Due to the methodology changes, results for SPEC Cloud IaaS 2018 are not comparable to those from the 2016 version.

Available for immediate download

The SPEC Cloud IaaS 2018 benchmark is available immediately for download from the SPEC website ( for $2,000. A $500 license is available for eligible academic and non-profit organizations. SPEC Cloud IaaS 2016 licensees will be offered a free upgrade to the 2018 version.

More information -- including a design overview, user guide, FAQs document, glossary, and run and reporting rules -- is available at

Published Thursday, December 20, 2018 7:05 AM by David Marshall
Filed under: ,
There are no comments for this post.
To post a comment, you must be a registered user. Registration is free and easy! Sign up now!
<December 2018>