chris - Fotolia

News Stay informed about the latest enterprise technology news and product updates.

NVMe over Fabrics gathers steam for flash and post-flash devices

Industry players are demonstrating the new NVM Express over Fabrics network interconnect technology, but it's hard to say when it will gain widespread adoption.

Products supporting Non-volatile Memory Express over Fabrics (NVMe-oF) are poised to start hitting the market,...

although across-the-stack support for the latency-lowering technology will be a work in progress for some time.

NVM Express Inc. -- an organization comprising more than 60 technology vendors -- in June released the 1.0 version of NVMe-oF, which enables the NVMe storage interface protocol to operate over Ethernet, Fibre Channel (FC), InfiniBand and other network fabrics.

NVMe is a higher-performance, lower-latency alternative to the Small Computer System Interface (SCSI) command set for transferring data between hosts and target storage devices. SCSI was created during the days of slower storage media, such as hard disk drives (HDDs) and tape. The NVMe specification, released in 2011, streamlines the register interface and command set for use with faster PCI Express-based solid-state drives (SSDs) and post-flash technologies, such as Intel-Micron's 3D XPoint.

"As we saw with SCSI, it lived for 30 years, and it still is living today," said Amber Huffman, an Intel fellow who serves as president at NVM Express Inc. "We tried to make sure we did not target NVMe just for flash because then it would be long in the tooth very quickly."

What we're trying to do is make sure that if you are across a fabric in a different part of the data center, that you can get your data as close to that raw latency of the NVMe PCIe SSD as possible.
Amber Huffmanpresident, NVM Express Inc.

The NVMe-oF specification defines an alternative transport for NVMe other than PCI Express. An NVM Express work group developed a remote direct memory access (RDMA) mapping for two variations of Ethernet -- RDMA over Converged Ethernet (RoCE) and internet Wide Area RDMA Protocol (iWARP) -- as well as InfiniBand. Work is also under way to support Intel's Omni-Path. The T11 Technical Committee responsible for FC standards hopes to finalize the FC-NVMe specification later this year.

The Fibre Channel Industry Association (FCIA) plans a technology demonstration of NVMe-oF using Fibre Channel storage networking technology at the Flash Memory Summit. The FCIA said it will use Brocade's latest G620 Gen 6 FC switch, Hewlett Packard Enterprise ProLiant DL380 servers, Broadcom Emulex LPe32002 host bus adapters (HBAs), and Toshiba PX04 NVMe drives to show a latency reduction of more than 55% compared to traditional SCSI.

Others planning NVMe-oF demonstrations at this week's Flash Memory Summit include Mellanox Technologies, which intends to show that NVMe-oF use will extend beyond Linux, and Kalray, which said it will collaborate with SK Hynix on a demo from compute node to remote NVMe SSDs.

NVMe-oF support still in the works

Despite the NVMe over Fabrics demonstrations by industry players, the technology is not ready for prime time use by the average enterprise IT organization. Drivers and optimizations for operating systems, hypervisors and adapters are still in the works.

One executive with a major storage vendor, who asked not to be identified, confirmed that IT end users probably won't see NVMe-oF supported in conventional flash storage systems before the second half of 2017. He expects a logical progression of components until that time.

"There is now a very early stage reference implementation on GitHub, but it's by no means something a customer would run for production," the storage executive said. "We're at the very early stages of a long process, which is off to a great start."

In June, the NVMe-oF working group published reference implementation code for host and target Linux drivers. But the Linux community is still working to add NVMe over Fabrics support to the Linux kernel, according to NVM Express sources. Support also remains a work in progress for Windows and VMware, they said.

Intel's Huffman said she expects NVMe-oF-compliant products to start to emerge later this year. She expects the first deployments of NVMe over Fabrics would use just a bunch of flash (JBOF) devices loaded with NVMe-based SSDs. Vendors could choose software or hardware to do the translation from NVMe over Fabrics on the front end to NVMe over PCIe on the back end, she noted.

Huffman said she has seen demonstrations where the read latency was less than a microsecond with a field-programmable gate array (FPGA) and less than 10 microseconds with software-based NVMe over Fabrics deployments. Current SCSI-based flash storage can add 100 microseconds of latency, she said.

"What we're trying to do is make sure that if you are across a fabric in a different part of the data center, that you can get your data as close to that raw latency of the NVMe PCIe SSD as possible," Huffman said.

Huffman noted the NVMe-oF work group's goal of adding no more than 10 microseconds of network latency. She said NVMe over Fabrics retains about 90% of the baseline NVMe specification.

Streamlined stack speeds performance

Dennis Martin, president and founder at Demartek LLC, an analyst organization that operates an on-site test lab in Golden, Colo., said individual NVMe devices exhibit noticeably higher performance and lower latency than traditional storage devices.

Martin attributed improvements in part to the streamlined NVMe software stack design, which requires fewer CPU instructions than other protocols to perform an I/O operation. He also noted that NVMe-oF is tailored for systems with multi-core processors, with its support for 64,000 I/O queues and up to 64,000 commands per queue.

"It will be a real game changer initially for high-end users who can't get enough throughput and IOPS with existing technologies and need the lowest possible latency for storage over distance," Martin wrote via email.

George Crump, founder and president at Storage Switzerland LLC, expects early NVMe over Fabrics deployments to focus on analytics processing and high-transaction databases. He said the impact would expand to larger numbers of users over time, although not every IT pro or data center will notice the difference.

"If your applications are responding instantly, how much more instant do you need?" Crump said. "This is really the first time I know of in data center history where we're providing more performance than the large majority of data centers can take advantage of. But as applications start to get geared into the fact that they have flash available to them at all times, I think that will change in three to five years."

Next Steps

NVMe-oF is among the storage tech innovations to watch

SSD storage using NVMe making strong inroads

Fabrics, fibre channel and other server-side tech explained

Dig Deeper on All-flash arrays