Written by Philipp Reisner, founder and CEO of LINBIT

NVMe over Fabrics (NVMe-oF) will be a key component in the future of storage environments. Why? It all began a few years ago when NAND flash started to make major inroads into the storage industry. The new storage medium needed to be accessed through existing interfaces like SATA and Serial attached SCSI (SAS). During that time, FusionIO created a NAND, flash-based SSD that was directly plugged into the PCIe slot of a server and came with a proprietary driver. This eliminated bottlenecks created by the interfaces from the era of rotating storage media, such as ATA or SCSI command sets.

Soon the industry created (within the SNIA body) an open standard, NVMe, that allows to connect fast storage directly to the PCIe bus.

However, the main problem for both SAS and SATA is that they can only transfer limited I/O currents to a device. A look at the geometry of a hard disk shows that it is difficult to impossible to process simultaneous I/O streams.

With a little goodwill, the read/write heads can process different requests together. One can also use buffering, but it doesn't scale.

Neither SAS nor SATA is designed to handle different I/O queues. AHCI had a queue depth of 32 commands. SCSI manages 128 to 256, depending on the implementation.

Individual queues have a negative effect on latency, which is worsened when data streams are larger. This issue is minimal with hard disks, but SSDs are effectively still slowed.

NVMe over Fabrics

NVMe is expected to be the solution to this problem, being the storage protocol for device connectivity. Similarly, NVMe-oF could replace iSCSI and Fibre Channel Protocols.

There are three variants for data transport, NVMe over Fabrics with Remote Direct Memory Access (RDMA) and NVMe over Fabrics with Fibre Channel (FC-NVMe) and NVMe/TCP.

RDMA allows data transfers to and from the application memory of two computers without using the processor. This allows low latency and high transfer speed. RDMA implementations are Infiniband, iWARP and RDMA over Converged Ethernet, also known as RoCE (pronounced "Rocky"). Some vendors, such as Mellanox, have adapter cards in their portfolio that enable speeds of 100 Gbps for both Infiniband and Ethernet, including NVMe over Fabrics Offload.

NVMe over Fibre Channel leverages advanced Fibre Channel technology which supports SCSI and NVMe storage transport with an upgrade.

What this means is that customers already using Fibre Channel can use the new technology with a simple firmware upgrade of their switches. Host bus adapters (HBAs) must support NVMe, typically 16 Gbps or 32 Gbps, and storage devices must be designed for NVMe over Fabrics.

NVMe/TCP uses a TCP stream as data transport. It compatible with existing ethernet infrastructures.

Adoption of NVMe

Of course, adopting new technologies is not a simple task, since it requires the upgrade of existing hardware. Studies have shown that only 6.5% of respondents connected more than 50% of their environment via NVMe.

Adoption of new technologies is also made difficult due to the complexity of migrating hardware and building comprehensive systems that benefit from the power of NVMe-based storage. Due to these reasons, only about half of respondents have not yet deployed NVMe.

However, there are several options available to IT departments who want to benefit from NVMe performance:

• One can use Parallel I/O technology to provide similar parallel processing benefits for non-NVMe hardware. It is fairly common to see a 5X improvement in storage performance when using Parallel I/O and additional intelligent caching algorithms to remove the I/O bottleneck created by single threading.

• With Software-Defined Storage (SDS) and dynamic block-level auto-tiering, even a small amount of NVMe storage can deliver significant performance improvements.

• Software-Defined Storage with Fibre Channel Gen6 support enables effective use of NVMe over a NAS - with up to 1.6 million IOPS over a port. This enables SDS to take full advantage of the performance potential of these systems and avoid migrations.
  

NVMe over Fabric is certainly the future-looking model for maximizing your performance. Right now it is built primarily for early adopters, but is expected to become mainstream in the near future. Of course, not every application requires NVMe performance. However, like most new technologies, many new storage systems are expected to be powered by NVMe in the near future.

Outlook

NVMe will replace SCSI and SAS as the standard connection for SSD drives. Therefore, for high-performance environments, the implementation of NVMe over Fabrics is justified.

It will be interesting to see which concept will prevail: either large-featured NVMe-oF targets with snapshots, replication, compression and deduplication in existing storage arrays, or lean architectures that just provide storage and rely on the OS or Application to do snapshots, replication, etc...

In the past, limited functionality prevented NVMe from making a breakthrough. However, NVMe will gradually replace older architectures that do not utilize the full potential of Flash.

To learn more about containerized infrastructure and cloud native technologies, consider coming to KubeCon + CloudNativeCon Barcelona, May 20-23 in Barcelona.

##

About the Author

 

Philipp Reisner is founder and CEO of LINBIT in Vienna, Austria. His professional career has been dominated by developing DRBD, a storage replication software for Linux. Today, he leads a company of roughly 30 employees with locations in Vienna, Austria and Portland, Oregon.