Virtualization Technology News and Information
How the NVMe-oF Protocol Redefines Kubernetes Storage

By Charles A. Piercey, Director of Product Management, KumoScale, KIOXIA America, Inc. 


Data storage that is locally-attached in data centers delivers low-latency performance as its primary benefit but is unable to maximize storage utilization.  On the other hand, network-attached storage historically has exhibited less than optimal latency performance but enables storage resource sharing for high storage utilization.  A solution that combines both, and with the ability to scale alongside Kubernetes, is available with the advent of the NVMe over Fabrics (NVMe-oF) specification.

A 'sea change' for data centers, the NVMe-oF specification requires a rethinking of the storage infrastructure from end-to-end because it enables a deployment approach called disaggregation.  Disaggregation is a form of shared storage where resources can be allocated to any server in the data center through a high-performance network.  It provides a number of benefits including the ability to:

  • Provision NVMebased storage with performance comparable to locally attached storage
  • Automate workload mobility, which is highly-prized by Kubernetes container orchestration
  • Automate a high-level of data resilience

Given these benefits, it is important to understand the backdrop to disaggregation, the core design of the NVMe-oF specification, how it relates to NVMe storage devices, and how this enables scalable data center storage services. 

The NVMe-oF specification will play a particularly important role in the future evolution of Kubernetes-managed infrastructures because legacy storage solutions inhibit these deployments by creating "gravity wells" around locally attached storage that reduce the workload flexibility needed by the Kubernetes scheduler.  Moving stateful applications from a locally-attached storage architecture to a shared storage model based on NVMe-oF technology can significantly reduce storage costs.  In fact, when the NVMe-oF specification is combined with an NVMe-oFnative SDS solution and the Kubernetes Container Storage Interface (CSI) driver, it empowers Kubernetes orchestration to scale data-intensive workloads both dynamically and efficiently.

NVMe / NVMe-oF Specification History

For decades, hard disk drives were the dominant storage media used in data centers worldwide.  The advent of solid state drives in 1999 was a major step forward in performance, utilizing flash memory cells to store data versus the seeks and searches associated with electro-magnetic hard drive storage.  Though SSDs demonstrated potential promise, their overall performance remained constrained by legacy hard drive storage protocols, interconnect latencies, and the ‘newness' and immaturity of flash memory device technology.  The high-end SSDs that did evolve used the PCI Express (PCIe) interface but were still using instructions and commands from non-standard protocols.  This created the need for a standardized protocol for all-flash storage.

The NVMe specification was introduced and provided a set of instructions and commands built from the ground up for flash-based storage media.  It utilizes the PCIe interface and locally-attached server slots to deliver fast SSD performance.  However, with advancements in scale-out applications and virtualization, directly connected server flash memory became a utilization bottleneck as locally-attached drives had either too much flash-based storage or not enough.  This forced customers to provision more storage than what was typically required, resulting in an underutilization of valuable resources and an increase in capital expenses.

The NVMe-oF specification was introduced to address these limitations, enabling servers to access remote NVMe-based SSDs over standard network protocols with about the same performance and latency of locally-attached drives.  The NVMe-oF specification can accomplish this because it uses Remote Direct Memory Access (RDMA) to efficiently access the memory of another computer without using the CPU or operating system of either.  This enables storage I/O to bypass the kernel stack entirely, delivering streamlined low-latency performance with minimal CPU consumption.

NVMe-oF Protocol Advantages for Data Center-Scale Storage

As the NVMe protocol replaces the older SCSI protocol, the NVMe-oF protocol replaces the older iSCSI and iSER network storage protocols and has several distinguishing technical attributes that add up to huge performance and scalability benefits for remote, shared storage, including:

  • Protocol Efficiency: The NVMe-oF protocol dramatically reduces the internal locking that is needed to serialize I/O, which in turn, improves interrupt handling efficiency.  It also supports message-signaled interrupts (MSI-X) and interrupt steering to prevent bottlenecks at the CPU level, enabling massive scalability.  When combined with the large number of cores available in modern processors, the NVMe-oF protocol delivers massive parallelism for connected devices, increasing I/O throughput while minimizing I/O latency.
  • Command Set Efficiency: The NVMe-oF protocol utilizes a streamlined and simple command set that requires less than half the number of CPU instructions to process one I/O request when compared to legacy protocols.  It also delivers higher input/output operations per second (IOPS) per CPU instruction cycle and lower I/O latency in the host software stack. 
  • Queue Parallelism: With 65,535 parallel queues, each with a queue depth of 65,535 entries, the NVMe-oF protocol supports massive I/O parallelism.  Each CPU has a private queue to communicate with a storage device and is capable of achieving high I/O speed because there are no locks between the queues that live on separate processing cores.  Since each controller has its own set of queues per CPU, I/O throughput increases linearly with the number of available processing cores.
  • Integrated Security: The NVMe-oF protocol supports secure tunneling features produced by the Trusted Computing Group (TCG) and other related security communities. 

Designed to solve the limitations of scale-out storage, the NVMe-oF specification lays the foundation for a storage fabric that dynamically shifts as workload demands shift, making it an ideal solution for the ever-changing world of Kubernetes-driven infrastructures.

Trends Driving Data Center Storage Disaggregation

New storage architectures are leveraging the NVMe-oF specification to connect compute and storage resources together flexibly and cost-effectively.  There are several supporting technological trends that enhance its value:

  • Fast Internal Server Bus: With the advent of PCIe Gen4, most servers can deliver 16 gigabytes per second (GB/s) of throughput internally, removing internal I/O bottlenecks.
  • Fast, High-Capacity Storage Media: A single NVMe-based SSD can deliver millions of IOPS, far exceeding the requirements of normal workloads running on a single compute server.  At the same time, SSD densities are up to 30.72 terabytes (TB) per drive, also exceeding what a single host CPU can effectively use.  These two trends increase the economic value of shared storage.
  • Fast Network: Data centers globally are replacing slower network connections with 100 or 200 Gigabit Ethernet (GbE) and faster networks that remove bandwidth limitations and bottlenecks.
  • Orchestrators Need for Storage Mobility: Kubernetes and virtual machine (VM) orchestrators need the mobility that shared storage delivers.  Otherwise, moving data-intensive storage volumes requires a high latency copy that limits the ability of orchestrators to optimize workload placement.

These trends combined with the NVMe-oF protocol should change how architects design and evaluate data center storage systems going forward.  The combination of high-density NVMe SSDs with the plummeting price of flash memory make storage dollars per IOPS, and storage read latency, the critical metrics for a data center storage architecture based on the NVMe-oF specification.

A Perfect Pairing: Kubernetes Orchestration and NVMe-oF Technology

Kubernetes container orchestration is a platform that helps build platforms and makes infrastructure fungible in a way that was inconceivable five years ago.  It delivers a highly flexible and efficient form of virtualization that transforms what used to be a hardware problem, into ‘infrastructure as code.' 

Traditional data centers built with locally-attached storage resources on each server make persistent data a challenge for Kubernetes-based applications.  Kubernetes orchestration uses lightweight container technology in place of VMs to manage workloads, and all of their dependencies, in an isolated state from other applications or system processes.  However, containers are more mobile than VMs and traditional storage infrastructures have had a hard time keeping pace.  While locally-attached storage provides good performance, when Kubernetes orchestration schedules a data-intensive container on another server, it forces a slow data storage copy operation that has inhibited container adoption.

The NVMe-oF specification is a major step forward for Kubernetes infrastructures because it enables shared storage across a network at an access latency similar to locally-attached storage.  Multiple NVMe-oF flash drives can be centralized into standard storage nodes with high-bandwidth network interfaces that serve up storage volumes to compute-only nodes running Kubernetes containerized workloads. 

The end result is that just the right amount of storage and compute can be allocated for each workload while delivering the high-performance, low-latency storage of locally-attached SSDs.  The centralization of storage resources adds a level of on-demand flexibility and cost-efficiency that ends up being critical to data infrastructures.  When a workload requires additional storage, volume allocation can be scaled on demand.  When application storage needs shrink or disappear entirely, the storage resources can be added back to the resource pool for other workloads to use. 


The NVMe protocol redefines the data center landscape by fundamentally reinventing storage I/O and extending that over standard network transports.  The result is a ‘sea change' to the way data centers are built and deployed.  Instead of spreading storage across compute nodes, flash devices can be disaggregated to shared storage servers that are deployed across failure zones and regions.  The results produce increased resource utilization and end-to-end data resilience. Companies need to aggressively evaluate and adopt NVMe-oF storage infrastructures in their data centers. More information available at


To learn more about containerized infrastructure and cloud native technologies, consider joining us at KubeCon + CloudNativeCon NA Virtual, November 17-20.

About the Author

Charles A. Piercey Director of Product Management, KumoScale, KIOXIA America, Inc. 

Charles Piercey 

Chuck Piercey is Director of Product Management at KIOXIA America, Inc. (formerly Toshiba Memory America, Inc.).  He is responsible for the KumoScale software defined storage platform.  A 40-year veteran of silicon valley and co-inventor on 5 patents, he has held previous product marketing/management positions with ROLM, IBM, Apple, SGI, ActaWorks, Business Objects, Trolltech, Nokia, TidalScale, and Juniper Networks.  He holds a BA from Stanford University and an MBA from the Anderson School at UCLA.

Published Thursday, October 29, 2020 11:56 AM by David Marshall
There are no comments for this post.
To post a comment, you must be a registered user. Registration is free and easy! Sign up now!
<October 2020>