BACKGROUND IMAGE: Sergey Nivens/stock.adobe.com

This content is part of the Essential Guide: How NVMe-oF advantages are transforming storage
Get started Bring yourself up to speed with our introductory content.

5 NVMe-oF implementation questions answered

Want to know more about implementing NVMe over fabrics, where it's useful and what to consider? Get answers to these and other questions surrounding this technology.

NVMe over fabrics is making its way into the enterprise, enabling organizations to extend NVMe's high-performance, low-latency advantages across their infrastructure.

An NVMe-oF implementation maps commands and responses between an NVMe host and shared memory, facilitating communication. It extends the distances over which NVMe devices can be accessed and enables scaling to multiple devices.

For most organizations, the first step into NVMe technology is an NVMe all-flash array using traditional network connections to the storage network. More than 60% of all-flash arrays will use NVMe storage media by 2022, according to G2M Research predictions. From there, a move to NVMe-oF may follow, depending on performance needs. G2M also estimated that more than 30% of AFAs will use NVMe-oF to connect to hosts or expansion shelves in the next three years.

NVMe-oF isn't without its complications and challenges, however. You must gather a lot of information and answer a number of questions before making the move. What follows is a look at five important NVMe-oF implementation questions to answer before deploying this new and critical technology.

What are the NVMe-oF implementation options?

When the initial NVMe-oF spec came out in 2016, it provided support for Fibre Channel (FC) and remote direct memory access (RDMA) fabrics, with InfiniBand, RDMA over Converged Ethernet and Internet Wide Area RDMA Protocol included under the RDMA umbrella. The NVMe-oF 1.1 spec, released in November, 2018, added TCP as a fabric option.

NVMe timeline
NVMe's significant milestones

With FC and RDMA, data can be transferred from a host to an array and back without being copied to memory buffers and I/O cards as must be done with a standard TCP/IP stack. This approach reduces the CPU load and data transfer latency, speeding up storage performance. However, FC and RDMA are more complex than TCP and can require special equipment and configurations when implementing NVMe-oF.

Enterprises can use TCP NVMe-oF transport binding over standard Ethernet networks, as well as the internet, making it simpler to implement. It doesn't require any reconfiguration changes or special equipment, eliminating many of the complications involved in using NVMe over FC or RDMA.

What factors must you consider when implementing NVMe-oF?

NVMe-oF is designed to use an organization's existing storage fabric, but that assumes the existing storage infrastructure is compatible with NVMe-oF. Some vendors don't strictly adhere to the NVMe-oF standard and, instead, use a proprietary approach that can lead to compatibility issues.

Another area to consider with NVMe-oF implementation is ensuring the storage infrastructure has adequate throughput. If the connectivity between the system's initiator and the target doesn't keep up with the physical NVMe storage device, system latency can increase. It's also important to check that storage system software is up to date. Old device drivers and OS kernels can cause issues.

Where does NVMe-oF leave rack-scale PCIe?

Before NVMe-oF appeared, it looked like data centers were headed toward rack-scale, switched PCIe fabric. DSSD, a company EMC acquired and then dismantled, developed technology that used the PCIe switch to share a block of flash across hosts in a rack. At the time, DSSD's 100 microsecond (µs) latency compared favorably to all-flash array vendors' 1 millisecond latency. 

But then NVMe-oF came on the scene, matching DSSD's 100 µs latency while letting customers use their existing standard Ethernet and FC networks, a development that shelved rack-scale PCIe for the time being. There's room for that to change, though, given the Gen-Z Consortium's focus on developing a rack-scale infrastructure that shares GPU and memory along with storage and I/O cards.

How does NVMe-oF help with scale-out storage?

Tools to scale-out a storage architecture use extra servers or nodes to increase its performance and capacity. In the past, scale-out tools tended to be complex to implement. But NVMe-oF is changing all that by skirting the limitations and complexities of traditional storage architecture.

NVMe-oF's advantage is in creating a more direct path between the host and the storage media. As a result, data isn't transmitted through a centralized controller. This more direct I/O path reduces the length of the path and enables a single host to talk to many drives and vice versa. This approach reduces storage system latency and increases scale-out capability.

Do NVMe and NVMe-oF solve all storage performance issues?

NVMe technology greatly enhanced performance between the server or storage controller CPU and attached flash SSDs over SAS and SATA SSDs. And NVMe-oF has reduced latency and improved the performance of shared storage. However, NVMe and NVMe-oF have also exposed another performance challenge: the CPU chokepoint in the server or the storage controller.

The latest processors support increasing numbers of SSDs. And as more drives are used, the supporting hardware gets increasingly complicated, with more CPUs, drives, drive drawers, switches, adapters, transceivers and cables for IT to cope with and manage. More hardware also results in diminishing marginal returns on overall performance.

The cause of this NVMe-oF implementation performance challenge is storage software not designed for CPU efficiency. One way to address this problem is to rewrite storage software to make it more efficient. Some vendors, such as StorOne, take that approach. Others are trying alternative solutions, such as adding more CPUs, putting dynamic RAM caching in front of NVMe SSDs and using computational storage to put more processors and RAM on NVMe drives.

Join the conversation

2 comments

Send me notifications when other members comment.

Please create a username to comment.

Where is your organization at in terms of implementing NVMe and NVMe over fabrics?
Cancel
Surely the comparison of FC (complex?) and TCP/IP is misleading and misses the whole point of investing in NVMe storage.
TCP/IP is not a network and requires a network such as Ethernet in order to function.It is unlikely that NVMe over TCP/IP would bring significant gains over SAS Flash based iSCSI storage.
The drive of NVMe is to replace the interrupt model developed around SCSI with a more CPU driven model which suits Flash memory and modern multi core CPUs. The performance gains are significant, sometimes in the order of 5 or 10 to 1. Fibre Channel is a very efficient, no drop, protocol which can provide the low latency, bandwidth and delivery hardware to keep these gains when extending the NVMe protocol outside the PCI bus. The RDMA function relieves the CPU of moving data between buffers and application memory which gain significantly reduces the overall latency.
While there are almost certainly use cases for NVMe over TCP/IP, especially when distance is concerned, the drive for better and faster storage is not one of them.
Cancel

-ADS BY GOOGLE

SearchDisasterRecovery

SearchDataBackup

SearchConvergedInfrastructure

Close