Flash! All-solid-state arrays

Solid-state storage has made its way into servers, hybrid storage arrays and caching devices, but all-solid-state arrays are an emerging class of high-performance storage systems.

Solid-state storage has made its way into servers, hybrid storage arrays and caching devices, but all-solid-state arrays are an emerging class of high-performance storage systems.

The more than 50-year reign of hard disk drives (HDDs) as the main location for primary data is being threatened. With falling prices and improved reliability, solid-state storage is starting to make inroads into the enterprise; and with the addition of solid-state products that complement HDD technology, a number of start-up companies have emerged offering all-flash array products.

Although solid-state drives (SSDs) have been deployed in traditional arrays for some years, high performance isn’t guaranteed by simply replacing HDD with SSD. As we’ll see, for best effect solid-state drives and flash memory need to be treated differently from their spinning disk counterparts.


The SSD difference

Solid-state drives combine flash memory and controller electronics to produce a device that operates similarly to a hard drive. Today, there are two types in use: single-level cell (SLC) and multi-level cell (MLC). SLC devices store a single bit of data in each cell, while MLC devices are capable of storing multiple (usually two) bits of data with each cell. SLC devices are more expensive per GB, but have a longer lifetime and are typically faster than their MLC counterparts. MLC devices are cheaper per GB, but have issues of endurance (the time before uncorrectable errors start appearing in read/write operations). Some vendors also cite a third category of flash memory, enterprise MLC (eMLC), which is a higher quality version of MLC with better resiliency and longer life.

Startups dominate all-flash array market

The main vendors in the all-flash array space are Kaminario Inc., Texas Memory Systems Inc., Violin Memory Inc. and Whiptail Technologies Inc. at the high end. Expect to pay $15/GB to $20/GB for up to 1 million IOPS per system. In the midrange, Pure Storage Inc. and SolidFire Inc. offer products with 100,000s of IOPS for approximately $5/GB to $10/GB, using data reduction techniques to achieve the lower price point.

Flash memory can be packaged to appear as a hard drive, emulating standard protocols such as SAS and SATA. Most vendors have opted to use these interfaces for their SSD implementations, but while the interface may be the same, SSDs and HDDs have very different characteristics.

Hard drives are mechanical in nature and their moving parts introduce latency into I/O operations. HDDs are particularly poor at handling highly random workloads due to the need to continually move the read/write heads, but they work better with sequential read and write operations.

SSDs have no moving parts and there’s no penalty with either sequential or random I/O, which makes them particularly well suited to certain workload types, such as virtual desktop infrastructure (VDI).

Solid-state drives are an order of magnitude faster than hard drives on both write and read transactions; a typical SSD drive is capable of more than 100,000 IOPS even on random I/O. SSDs are also lighter and use significantly less power than hard disk drives, which can have a bearing on the total cost of ownership (TCO) of storage arrays for many organizations.


The quirks of SSD

Despite all their apparent benefits, SSDs aren’t perfect devices and they have their own peculiarities. Most significantly, flash memory has a finite write lifetime, so SSDs eventually wear out. Typically, SLC flash endures approximately 100,000 write cycles; MLC flash is an order of magnitude worse at around 10,000 cycles for individual data cells. Past those points, storing and retrieving data becomes unreliable. Manufacturers use a number of techniques to extend life, including wear leveling, error correction codes, bad block remapping and overprovisioning.

SSDs may also exhibit inconsistent response times. Data written to an SSD is spread across the entire device to ensure even wearing of individual cells. This process is known as wear leveling. As new data is written, blocks of cells are marked for garbage collection. As an SSD reaches a high level of utilization or write activity, the process of garbage collection can cause spikes in response times as invalid data areas are recovered. Clearly, this kind of behavior is undesirable in low-latency environments.

To emulate a hard drive or not to

Packaging flash memory in the same form factor as a traditional hard disk offers obvious benefits. It can be integrated into an array simply by replacing an existing hard drive and it’s compatible with existing protocols and form factors. Most SSD array vendors have taken this option, using readily available SSDs as their standard form factor. However, two vendors have chosen another route.

Nimbus Data Systems Inc. designed its own SSD devices, which they call enterprise flash modules (EFMs). Based on eMLC, these devices are custom-made to work within Nimbus’ S-Class or E-Class storage arrays. The EFMs have dedicated controller software and additional local nonvolatile RAM (NVRAM) that enables writes to be confirmed once completed to this faster RAM rather than waiting to have the data committed to the flash memory itself.

Violin Memory Inc. uses its own custom Violin Intelligent Memory Modules (VIMMs). These devices combine flash memory and controller logic with a three-port switch that provides redundant connections to other VIMMs in an array for high availability.


Designing for SSD

As noted earlier, some array vendors took the step of offering all-SSD versions of their traditional storage arrays. Although these devices are faster than their HDD counterparts, they don’t provide the same relative increase in performance that can be achieved with arrays specifically designed for solid-state drives. Traditional storage arrays were designed around the assumption that the hard drive was the slowest part of the infrastructure, so the designs implemented array-based caching and complex algorithms to place data and pre-fetch data from spinning disks. These features have no benefits when using SSD; the order of magnitude increase in SSD performance requires new architectural thinking.

One good example of this is the removal of cache from SSD array controllers. Whiptail Technologies Inc. doesn’t use any cache in its controllers, and Pure Storage Inc., another all-flash array vendor, has pushed the cache to the disk shelf. Nimbus and Violin have moved NVRAM cache to the flash module. Removing controller cache means no additional logic is required for cache coherency between controllers in high-availability configurations. This simplifies design and increases performance.

Traditional storage arrays have only recently moved to using all-SAS back-end connectivity. Fast 6 Gbps connectivity for all hard drives wasn’t essential, as traditional hard drives couldn’t fully saturate a SAS connection. However, SSDs are easily capable of using full SAS speeds, so all-SSD arrays need full SAS bandwidth to all back-end devices.

In addition to the wear-leveling technology implemented within SSDs to ensure even usage of all memory cells and a longer, effective life, all-SSD array vendors implement array-level wear leveling, ensuring I/O is balanced across all SSDs or flash modules to both optimize IOPS and increase the longevity of the flash.

Many vendors also make use of compression and data deduplication techniques to increase the effective capacity of their arrays. Deduplication works well with SSDs as there’s no performance penalty from the increased randomness of I/O that dedupe introduces. Pure Storage, for example, uses a range of data reduction techniques in its products, as well as variable RAID stripes and dynamic RAID rebuilds to deliver consistent I/O performance.


Use cases for all-flash arrays

Traditional wisdom over the last few years has promoted the idea of tiering data across multiple layers of storage -- an approach that all-SSD arrays seem to contradict. However, the tiering methodology arose because storage arrays have grown and been used to serve general-purpose data. Solid-state arrays aren’t yet as highly scalable as traditional arrays, and it’s obvious that the relative expense of solid-state drives means all-flash arrays will be targeted at high IOPS applications rather than being used as general-purpose storage.

Typically, SSD arrays are being positioned as suitable for applications where consistent and low latency is a requirement. This includes databases, virtual desktops, gaming websites and financial trading systems.


The traditional $/GB metric no longer applies for all-flash arrays; instead IOPS/$ and IOPS/GB are more effective measurements when comparing vendors. IOPS/$ indicates the cost effectiveness of the device when looking to get the best “bang for the buck” in high I/O environments. IOPS/GB indicates the relative I/O density of an array -- the ability to do as many reads and writes as possible per GB of storage.

We’re already seeing a division of SSD array vendors into those at the high end, delivering 500,000 to 1 million IOPS with sub-millisecond response times. These vendors include Kaminario, Texas Memory Systems, Violin Memory and Whiptail. Prices are typically around $15/GB to $20/GB.

Mainstream arrays from vendors such as Pure Storage and SolidFire Inc. deliver performance in the 100,000s of IOPS, with response times in the 1ms to 5ms range. Prices are lower than those of high-end products at approximately $5/GB to $10/GB.

New metrics

The $/GB metric has been used for many years as a relative measure of array cost, especially as hard drive prices have eroded and drive capacities increased. However, $/GB isn’t a truly effective measurement when comparing high-end, all-flash arrays as the aim isn’t to deliver general-purpose, centralized storage but rather to meet specific application requirements. Where latency and IOPS are key, two measurements are more appropriate:

•    IOPS/$, which measures “bang for the buck”

•    IOPS/GB, which measures the I/O density of the storage array

Midrange vendors, such as Pure Storage and SolidFire, have chosen to implement data reduction techniques, such as compression and data deduplication, to increase the effective capacities of their SSD arrays and be in a position to compete on a more equal footing with traditional arrays in the $/GB battle. But not all data compresses equally; virtual server and virtual desktop data will compress well, but database data may not, so the $/GB figure quoted by these vendors may have to be taken as more of an indicator rather than an actual value.

No flash in the pan

All-flash storage arrays are a new class of product delivering high volumes of IOPS for demanding applications. We’re already seeing a segmentation of the market into mainstream and high-end products, based on IOPS and price. While not for everyone, all-flash arrays will find a niche in most enterprise environments.

BIO: Chris Evans is a UK-based storage consultant. He maintains The Storage Architect blog.

Dig Deeper on All-flash arrays