By Charles A. Piercey, Director of Product Management,
KumoScale, KIOXIA America, Inc.
Introduction
Data
storage that is locally-attached in data centers delivers low-latency
performance as its primary benefit but is unable to maximize storage
utilization. On the other hand,
network-attached storage historically has exhibited less than optimal latency
performance but enables storage resource sharing for high storage
utilization. A solution that combines
both, and with the ability to scale alongside Kubernetes, is available with the
advent of the NVMe over Fabrics (NVMe-oF) specification.
A 'sea change'
for data centers, the NVMe-oF specification requires a rethinking of the
storage infrastructure from end-to-end because it enables a deployment approach
called disaggregation. Disaggregation is
a form of shared storage where resources can be allocated to any server in the
data center through a high-performance network.
It provides a number of benefits including the ability to:
-
Provision NVMebased storage with
performance comparable to locally attached storage
-
Automate workload mobility, which is
highly-prized by Kubernetes container orchestration
-
Automate a high-level of data
resilience
Given these
benefits, it is important to understand the backdrop to disaggregation, the
core design of the NVMe-oF specification, how it relates to NVMe storage
devices, and how this enables scalable data center storage services.
The NVMe-oF
specification will play a particularly important role in the future evolution
of Kubernetes-managed infrastructures because legacy storage solutions inhibit
these deployments by creating "gravity wells" around locally attached storage
that reduce the workload flexibility needed by the Kubernetes scheduler. Moving stateful applications from a
locally-attached storage architecture to a shared storage model based on
NVMe-oF technology can significantly reduce storage costs. In fact, when the NVMe-oF specification is combined
with an NVMe-oFnative SDS solution and the Kubernetes Container Storage Interface (CSI) driver, it empowers Kubernetes
orchestration to scale data-intensive workloads both dynamically and efficiently.
NVMe
/ NVMe-oF Specification History
For decades, hard
disk drives were the dominant storage media used in data centers
worldwide. The advent of solid state
drives in 1999 was a major step forward in performance, utilizing flash memory
cells to store data versus the seeks and searches associated with
electro-magnetic hard drive storage.
Though SSDs demonstrated potential promise, their overall performance
remained constrained by legacy hard drive storage protocols, interconnect
latencies, and the ‘newness' and immaturity of flash memory device
technology. The high-end SSDs that did
evolve used the PCI Express (PCIe) interface but were still using
instructions and commands from non-standard protocols. This created the need for a standardized
protocol for all-flash storage.
The NVMe
specification was introduced
and provided a set of instructions and commands built from the ground up for
flash-based storage media. It utilizes
the PCIe interface and locally-attached server slots to deliver fast SSD performance. However, with advancements in scale-out
applications and virtualization, directly connected server flash memory became
a utilization bottleneck as locally-attached drives had either too much
flash-based storage or not enough. This
forced customers to provision more storage than what was typically required,
resulting in an underutilization of valuable resources and an increase in
capital expenses.
The NVMe-oF
specification was introduced to address these limitations, enabling servers to
access remote NVMe-based SSDs over standard network protocols with about the
same performance and latency of locally-attached drives. The NVMe-oF specification can accomplish this
because it uses Remote Direct Memory Access (RDMA) to efficiently access the
memory of another computer without using the CPU or operating system of
either. This enables storage I/O to
bypass the kernel stack entirely, delivering streamlined low-latency
performance with minimal CPU consumption.
NVMe-oF
Protocol Advantages for Data Center-Scale Storage
As the NVMe
protocol replaces the older SCSI protocol, the NVMe-oF protocol replaces the
older iSCSI and iSER network storage protocols and has
several distinguishing technical attributes that add up to huge performance and
scalability benefits for remote, shared storage, including:
- Protocol Efficiency: The NVMe-oF protocol dramatically
reduces the internal locking that is needed to serialize I/O, which in turn,
improves interrupt handling efficiency.
It also supports message-signaled interrupts (MSI-X) and interrupt
steering to prevent bottlenecks at the CPU level, enabling massive
scalability. When combined with the
large number of cores available in modern processors, the NVMe-oF protocol
delivers massive parallelism for connected devices, increasing I/O throughput
while minimizing I/O latency.
- Command Set Efficiency: The NVMe-oF protocol utilizes a
streamlined and simple command set that requires less than half the number of
CPU instructions to process one I/O request when compared to legacy
protocols. It also delivers higher
input/output operations per second (IOPS) per CPU instruction cycle and lower
I/O latency in the host software stack.
- Queue Parallelism: With 65,535 parallel queues, each with
a queue depth of 65,535 entries, the NVMe-oF protocol supports massive I/O
parallelism. Each CPU has a private
queue to communicate with a storage device and is capable of achieving high I/O
speed because there are no locks between the queues that live on separate
processing cores. Since each controller
has its own set of queues per CPU, I/O throughput increases linearly with the
number of available processing cores.
- Integrated Security: The NVMe-oF protocol supports secure
tunneling features produced by the Trusted Computing Group (TCG) and other
related security communities.
Designed to solve
the limitations of scale-out storage, the NVMe-oF specification lays the
foundation for a storage fabric that dynamically shifts as workload demands
shift, making it an ideal solution for the ever-changing world of
Kubernetes-driven infrastructures.
Trends
Driving Data Center Storage Disaggregation
New
storage architectures are leveraging the NVMe-oF specification to connect
compute and storage resources together flexibly and cost-effectively. There are several supporting technological
trends that enhance its value:
-
Fast Internal Server Bus: With the advent of PCIe Gen4, most servers can deliver 16
gigabytes per second (GB/s) of throughput internally, removing internal I/O
bottlenecks.
-
Fast, High-Capacity Storage Media: A single NVMe-based SSD can deliver millions of IOPS, far
exceeding the requirements of normal workloads running on a single compute
server. At the same time, SSD densities
are up to 30.72 terabytes (TB) per drive, also exceeding what a
single host CPU can effectively use.
These two trends increase the economic value of shared storage.
-
Fast Network: Data centers globally are replacing slower network connections
with 100 or 200 Gigabit Ethernet (GbE) and faster networks that remove
bandwidth limitations and bottlenecks.
-
Orchestrators Need for Storage Mobility: Kubernetes and virtual machine (VM) orchestrators need the
mobility that shared storage delivers. Otherwise,
moving data-intensive storage volumes requires a high latency copy that limits
the ability of orchestrators to optimize workload placement.
These trends combined with the
NVMe-oF protocol should change how architects design and evaluate data center
storage systems going forward. The
combination of high-density NVMe SSDs with the plummeting price of flash memory
make storage dollars per IOPS, and storage read latency, the critical metrics
for a data center storage architecture based on the NVMe-oF specification.
A
Perfect Pairing: Kubernetes Orchestration and NVMe-oF Technology
Kubernetes container
orchestration is a platform that helps build platforms and makes infrastructure
fungible in a way that was inconceivable five years ago. It delivers a highly flexible and efficient
form of virtualization that transforms what used to be a hardware problem, into
‘infrastructure as code.'
Traditional data centers
built with locally-attached storage resources on each server make persistent
data a challenge for Kubernetes-based applications. Kubernetes orchestration uses lightweight
container technology in place of VMs to manage workloads, and all of their
dependencies, in an isolated state from other applications or system
processes. However, containers are more
mobile than VMs and traditional storage infrastructures have had a hard time
keeping pace. While locally-attached
storage provides good performance, when Kubernetes orchestration schedules a
data-intensive container on another server, it forces a slow data storage copy
operation that has inhibited container adoption.
The NVMe-oF specification is a
major step forward for Kubernetes infrastructures because it enables shared
storage across a network at an access latency similar to locally-attached
storage. Multiple NVMe-oF flash drives
can be centralized into standard storage nodes with high-bandwidth network
interfaces that serve up storage volumes to compute-only nodes running
Kubernetes containerized workloads.
The end result is that just
the right amount of storage and compute can be allocated for each workload
while delivering the high-performance, low-latency storage of locally-attached
SSDs. The centralization of storage
resources adds a level of on-demand flexibility and cost-efficiency that ends
up being critical to data infrastructures.
When a workload requires additional storage, volume allocation can be
scaled on demand. When application
storage needs shrink or disappear entirely, the storage resources can be added
back to the resource pool for other workloads to use.
Summary
The
NVMe protocol redefines the data center landscape by fundamentally reinventing
storage I/O and extending that over standard network transports. The result is a ‘sea change' to the way data
centers are built and deployed. Instead
of spreading storage across compute nodes, flash devices can be disaggregated
to shared storage servers that are deployed across failure zones and
regions. The results produce increased
resource utilization and end-to-end data resilience. Companies need to
aggressively evaluate and adopt NVMe-oF storage infrastructures in their data
centers. More information available at https://kumoscale.kioxia.com/.
##
To learn more about containerized infrastructure and cloud native technologies, consider joining us at KubeCon + CloudNativeCon NA Virtual, November 17-20.
About the Author
Charles A. Piercey Director of Product Management,
KumoScale, KIOXIA America, Inc.
Chuck Piercey is Director of Product Management at KIOXIA
America, Inc. (formerly Toshiba Memory America, Inc.). He is responsible
for the KumoScale software defined storage platform. A 40-year veteran of
silicon valley and co-inventor on 5 patents, he has held previous product
marketing/management positions with ROLM, IBM, Apple, SGI, ActaWorks, Business
Objects, Trolltech, Nokia, TidalScale, and Juniper Networks. He holds a
BA from Stanford University and an MBA from the Anderson School at UCLA.