NVMe (non-volatile memory express)

Contributor(s): Carol Sliwa

NVMe (non-volatile memory express) is a host controller interface and storage protocol created to accelerate the transfer of data between enterprise and client systems and solid-state drives (SSDs) over a computer's high-speed Peripheral Component Interconnect Express (PCIe) bus.

The NVMe specification defines a register interface, command set and collection of features for PCIe-based SSDs with the goals of high performance and interoperability across a broad range of NVM subsystems. The NVMe specification does not stipulate the ultimate usage model, such as solid-state storage, main memory, cache memory or backup memory.

NVMe provides an alternative to the Small Computer System Interface (SCSI) standard and the Advanced Technology Attachment (ATA) standard for connecting and transmitting data between a host system and a peripheral target storage device. The ATA command set in use with Serial ATA (SATA) SSDs and the SCSI command set for Serial Attached SCSI (SAS) SSDs were developed at a time when hard disk drives (HDDs) and tape were the primary storage media. NVMe was designed for use with faster media.

The main benefits of NVMe-based PCIe SSDs over SAS-based and SATA-based SSDs are reduced latency in the host software stack, higher input/output operations per second (IOPS), and potentially lower power consumption, depending on the form factor and the number of PCIe lanes in use.

David Akerson, Intel's platform
marketing engineer, explains

NVMe can support SSDs that use different types of non-volatile memory, including NAND flash and the 3D XPoint technology developed by Intel and Micron Technology. Supported form factors include add-in PCIe cards, M.2 SSDs and U.2 2.5-inch SSDs. NVMe reference drivers are available for a variety of operating systems, including Windows and Linux.

How NVMe works

NVMe maps I/O commands and responses to shared memory in a host computer over the PCIe interface. The NVMe interface supports parallel I/O with multicore processors to facilitate high throughput and mitigate central processing unit (CPU) bottlenecks.

NVMe offers a more streamlined command set to process an I/O request than the SCSI and ATA command sets do. NVMe requires fewer than half the number of CPU instructions than the SCSI command set does with SAS devices and the ATA command set uses with SATA drives.

NVMe supports 64,000 commands in a single message queue and a maximum of 65,535 I/O queues. By contrast, a SAS device typically supports up to 256 commands, and a SATA drive supports up to 32 commands, in one queue.

However, NVMe-based PCIe SSDs are currently more expensive than SAS- and SATA-based SSDs of equivalent capacity, and high-end enterprise NVMe SSDs may consume more power than SAS or SAS SSDs. The SCSI Trade Association claims the more mature SAS SSDs offer additional advantages over NVMe PCIe SSDs, such as greater scalability, hot pluggability and time-tested failover capabilities. NVMe PCIe SSDs also may provide a level of performance that many applications do not require.

History of NVM Express

The Non-Volatile Memory Host Controller Interface (NVMHCI) Work Group began to develop the NVMe specification in 2009 and published the 1.0 version on March 1, 2011. The 1.0 specification included the queueing interface, the NVM command set, administration command set and security features.

The NVMHCI Work Group, commonly known as the NVM Express Work Group, released an update to the NVMe specification on Oct. 11, 2012. NVMe 1.1 added support for SSDs with multiple PCIe ports to enable multipath I/O and namespace sharing. Other new capabilities included autonomous power state transitions during idle time to reduce energy needs and reservations allowing two or more hosts to coordinate access to a shared namespace to improve fault tolerance.

NVMe specification timeline

The NVM Express Work Group held its first plugfest in May 2013 to enable companies to test its products' compliance to the NVMe specification and to check interoperability with other NVMe products.

The NVMe 1.2 specification emerged on Nov. 3, 2014, with enhancements such as support for live firmware updates, improved power management and the option for end-to-end data protection.

On Nov. 17, 2015, the NVM Express organization ratified the 1.0 version of the NVM Express Management Interface (NVMe-MI) to provide an architecture and command set to manage a non-volatile memory subsystem out of band. NVMe-MI enables a management controller to perform tasks such as SSD device and capability discovery, health and temperature monitoring, and nondisruptive firmware updates. Without NVMe-MI, IT managers generally relied on proprietary, vendor-specific management interfaces to enable administration of PCIe SSDs.

NVMe 1.3 feature enhancements

NVM Express released NVMe 1.3 in June 2017. Highlights center on sanitize, a new framework known as directives, and virtualization enhancements.

In a sanitize operation, all user data in the NVMe subsystem is modified so that recovery is not possible "from any cache, nonvolatile media or controller memory buffer," according to an NVM Express reference sheet. Sanitize operations are recommended when an SSD is being retired or reused for a new use case. Sanitize modes include low-level block erase on NAND media, crypto-erase to change a media encryption key, and overwrite.

The Directives framework defines a mechanism for the exchange of data between a host and an NVMe subsystem. This enables per-I/O command tagging and gives IT administrators the ability to configure reportable attributes and settings.

The first use of Directives is a feature called Streams for optimizing data placement to boost the endurance and performance of NAND SSDs. Traditionally, before new data can be written to the SSD, large blocks of data first must be erased.

The Streams feature enables a host to use a "stream identifier" to indicate the specific logical blocks of storage that belong to a group of associated data. This enables a read or a write to be tagged with related data stored in other locations.

Virtualization enhancements define how NVMe flash could be used in a shared storage environment where both physical and virtual controllers are present, including primary storage controllers and secondary storage controllers. NVM Express said the goal is to enable development teams to dedicate a specific SSD to a specific virtual machine.

The next NVMe version is expected in 2018 or 2019. Roadmap items include improved NVMe SSD I/O determinism, persistent control memory buffer support and SAN multipathing. Projected NVMe over Fabrics roadmap items include authentication, enhanced discovery and Transmission Control Protocol transport support.

NVM Express Inc.

The NVM Express Work Group incorporated under the name NVM Express Organization in March 2014. Founding members at the time included Cisco Systems, Dell, EMC, Western Digital's HGST subsidiary, Intel, LSI, Micron Technology, NetApp, Oracle, PMC-Sierra, Samsung Electronics, SanDisk and Seagate Technology.

The NVM Express Organization later became known simply as NVM Express Inc. The nonprofit organization has more than 100 member technology companies.

NVM Express Inc. has a 13-member board of directors to govern the organization and set its strategic direction. Six board members serve multiyear terms, and the remaining seven board members are elected on an annual basis from the NVMe Express membership. Board members are alternately referred to as promoter companies.

Board members with multiyear terms expiring in 2017 are Cisco, Dell EMC, Intel, Microsemi, NetApp and Oracle. Board members elected to single-year terms for 2017 are Facebook, Micron, Microsoft, Samsung, Seagate, Toshiba and Western Digital.

NVMe over Fabrics

NVM Express Inc. published the 1.0 version of the NVMe over Fabrics (NVMe-oF) specification on June 5, 2016. NVMe-oF is designed to extend the high-performance and low-latency benefits of NVMe across network fabrics that connect servers and storage systems, such as Fibre Channel (FC), Ethernet and InfiniBand.

Fabric transports include NVM-oF using remote direct memory access (RDMA) and NVMe-oF mapped to FC. A technical subgroup of NVM Express Inc. worked on NVMe-oF with RDMA, and the T11 committee of the International Committee for Information Technology Standards (INCITS) is responsible for the development of FC-NVMe.

Listen in

Demartek Founder Dennis Martin explains the importance of the NVMe and NVMe over Fabrics protocols.

The NVMe-oF specification is largely the same as the NVMe specification. One of the main differences between NVMe-oF and NVMe is the methodology for transmitting and receiving commands and responses. NVMe is designed for local use and maps commands and responses to a computer's shared memory via PCIe. By contrast, NVMe over Fabrics employs a message-based system to communicate between the host computer and target storage device.

How NVMe-oF Works

The stated design goal for NVMe-oF was to add no more than 10 microseconds of latency for communication between an NVMe host computer and a network-connected NVMe storage device, in comparison to the latency associated with an NVMe storage device using a local computer's PCIe bus.

See also: non-volatile storage

This was last updated in May 2017

Continue Reading About NVMe (non-volatile memory express)

Dig Deeper on Solid-state storage

Join the conversation


Send me notifications when other members comment.

Please create a username to comment.

What are the impediments to NVMe adoption?
As I see and hear, seems the Intel Optrane memory spec, SCM, M2, power fail safety, 32GB Fiber Channel  all need to be resolved before main stream. I can hardly wait to put a 20u second latency storage device under my SQL ERP.
Clear and easy to understand overview of the technology