Four major developments have dominated the flash storage landscape in recent years: 3D architectures allow for...
denser NAND chips; memory channel flash uses a server's DIMM slots and bus for additional solid-state storage; nonvolatile memory express (NVMe) delivers a more efficient means of tapping into a server's PCI bus to extract greater performance; and NVDIMM tacks on a nonvolatile tier to system RAM.
All of these technologies are available now -- in this article, you'll find out the best use cases and the advantages/disadvantages of each. In addition, we will take a look at promising technologies that could replace NAND flash technology.
NAND flash usage has seen tremendous acceptance and growth, from consumer devices to the desktop and into the data center. NAND flash technology is used for primary (and the only) storage in many consumer devices, and it is also deployed in various forms with and without hard disk technology on the desktop and in the data center.
Until recently, most of this NAND flash has been two-dimensional "planar" NAND flash. Over the last few years, NAND flash technology advancements have been primarily described by changes in the lithography (or die size, measured in nanometers), density (gigabits per square millimeter), bits per cell (single, multi- and triple-level cell) and power consumption. In addition to the technology advancements, the overriding factor is, and will continue to be, the cost to produce it.
Although all the NAND flash manufacturers have roadmaps that indicate further 2D planar NAND flash development, they also have 3D NAND roadmaps. A relatively new form of NAND flash memory, 3D NAND is built in three dimensions, stacking several layers in the "vertical" dimension. Vertical stacking improves the performance, reliability and capacity of the devices because it relaxes some of the technical constraints while increasing capacity. The new key technical parameter for 3D NAND is the number of layers. Today's 3D NAND chips include 24 and 32 layers. Roadmaps show the number of layers increasing to 48 or 64 in the next two or three years.
Some solid-state drives (SSDs) based on 3D NAND technology are available today, and more are expected to follow. When any memory technology emerges, the initial production costs are high and yields are low. It requires a year or more of production for the yields to increase and the volumes to grow enough to bring the costs down to a level that make it both worthwhile to produce and for the market to accept. Industry estimates are that it will probably take two or three years before 3D NAND production can move into high gear.
While the NAND flash manufacturers continue to make design and production improvements to both 2D and 3D NAND flash technology, other companies are inventing new ways to deploy this nonvolatile memory. The goal for some of these ideas is to move this technology closer to the processor. As more technologies move in this direction, the distinction between storage and memory becomes increasingly blurred. These next technologies are various implementations of putting nonvolatile memory onto the memory channel.
NVMe is a new storage protocol designed specifically to work with nonvolatile memory types such as NAND flash that are used for storage devices. It is a streamlined protocol that can handle very large queue depths and numbers of queues for these fast devices. One of the long-range concerns is that, as the device types continue to become faster and faster, the storage software stacks in the operating systems will become the bottleneck in terms of latency rather than the hardware. NVMe is aimed at avoiding that issue.
NVMe currently uses the PCI Express bus as the physical interface to NVMe devices, which are available in multiple form factors. The first set of NVMe devices to be released were enterprise-class SSDs, but client (consumer) NVMe SSDs are expected to emerge. In our lab tests, we have seen that NVMe SSDs are quite a bit faster than SATA or SAS SSDs and have much lower latency.
The NVM Express organization recently announced "NVMe over Fabrics" in order to extend the capabilities of NVMe over distance and that can support many devices. This will be a very interesting area of development for the rest of 2015 and into 2016.
NVDIMM technology is one way to put nonvolatile memory (NVM) on the memory channel via DIMM slots. NVDIMM technology includes DRAM, NVM (currently NAND flash) or both. It can be grouped into three different types: NVDIMM-N, NVDIMM-F and NVDIMM-P. Most of these technologies require some BIOS changes and may require operating system changes to fully support them.
NVDIMM-N is a DIMM that contains both DRAM and NVM, but only the DRAM is visible to the system, appearing as a standard RDIMM with typical DRAM capacity and latency. The NVM is not addressable by the host server and simply acts as a backup for the DRAM. There is at least as much NVM as DRAM on the DIMM, so that all the data in the DRAM can be protected. NVDIMM-N behaves the same as normal DRAM, except that the DRAM contents are saved to the NVM in the event of a power loss with supercapacitors providing power to copy the DRAM to the NVM. BIOS support for NVDIMM-N is required to perform the data protection functions.
NVDIMM-F has only NVM and no DRAM and is sometimes known as "memory channel flash." It uses block-oriented access to the storage and can map the NVM into the memory address space. The NVM capacities are typical of SSDs, but the latencies are much lower than regular storage -- typically in the single or double-digit microseconds.
NVDIMM-P includes both DRAM and NVM and combines the functions of NVDIMM-N and NVDIMM-F onto the same module. The NVM is allocated into two areas, one to provide persistence for the DRAM and the other that is accessible as block storage.
Use cases for emerging flash technologies
There are a number of use cases in the enterprise for these new types of storage. Cloud computing, virtualization, high-performance computing (HPC), databases, big data, real-time video editing of 4K content are a few that come to mind.
These faster storage devices and persistent memory devices can deliver higher overall performance and lower latency which means that decisions can be made more quickly -- often in real-time. Applications respond faster and more consistently, delivering better quality of service (QoS). These characteristics can also help improve virtual machine utilization and scalability.
There are some interesting side effects, however. When storage has increased performance and significantly lower latency, CPU utilization can go up significantly, because the CPU is not waiting as long for I/O operations to complete and therefore gets more work done in less time. This means that your physical-to-virtual server ratio might change if you base the number of virtual machines you deploy per physical host on CPU utilization. We have also seen very fast storage uncover bottlenecks in other areas, such as networking.
I have been saying for some time that solid-state storage and faster networks were made for each other.
Persistent memory implications
Many of these new nonvolatile memory technologies are classified as "Persistent Memory" (PM) technology. This new PM technology will require that application developers re-think how to write applications and other major software components, such as file systems and operating systems.
Persistent memory appears as byte-addressable memory to applications, not storage, yet because of its persistence, can be expected to be available for retrieval from the same logical location at a much later time. The notion of a "LUN" and "file" must be changed then, to adapt to a new technology that maps memory addresses with what used to be considered a storage device.
Many questions arise with this technology. Will we still need drivers to address storage? Will file systems need to be made PM-aware? Will we still need RAID or its equivalent?
As you might imagine, some new programming models are required to fully exploit these technologies. The Storage Networking Industry Association (SNIA) approved version 1.1 of the NVM programming model (NPM) in March 2015. This specification addresses some of the technical issues relating to persistent memory that are different from the classic ways we address storage.
Several industry groups are addressing the impact of persistent memory on future software systems. If you are interested in exploring further into this topic, you will find the SNIA NVM Programming Technical Workgroup and the Persistent Memory Programming project a good place to start. Both industry groups provide in-depth technical detail for these software development topics.
New memory types
The underlying technology that allows NAND flash to store data is based on a floating gate transistor. Some new memory technologies are based on different underlying technologies that are beginning to emerge as possible future successors to NAND flash technology. These include Phase Change Memory, Resistive RAM including Memristors, Magneto-resistive RAM including spin-transfer torque, and a few others.
Some or all of these may become available in the next two to five years in noticeable quantities. The cost along with capacity and speed of these technologies will influence which, if any, will succeed NAND flash technology.
In July 2015, Intel and Micron jointly announced a "breakthrough" new memory technology called 3D XPoint technology. They claim that this "technology is up to 1,000 times faster and has up to 1,000 times greater endurance than NAND, and is 10 times denser than conventional memory." This certainly has the potential of filling a gap between today's DRAM and today's NAND flash memory.
Will NAND flash memory stick around?
The future's bright for DIMM, perhaps more than PCIe NAND flash
Memory channel uses NAND to benefit flash storage