Big data analytics will place new burdens on data storage systems. Here are some of the key features those systems will need to meet the challenges of big data.
"Big data" refers to data sets that are too large to be captured, handled, analyzed or stored in an appropriate timeframe using traditional infrastructures. Big is, of course, a term relative to the size of the organization and, more importantly, to the scope of the IT infrastructure that’s in place. Big data also infers analysis, driven by the expectation that there’s value in all the information businesses are accumulating -- if there was just a way to pull that value out.
Perhaps it follows from the notion that storage capacity is cheap, but in the effort to cull business intelligence from the mountains of new data created every day, organizations are saving more of it. They’re also saving data that’s already been analyzed, which could potentially be used for trending against future data collections.
Why big, why now?
Aside from the ability to keep more data than ever before, we have access to more types of data. These data sources include Internet transactions, social networking activity, automated sensors, mobile devices and scientific instrumentation, among others. In addition to static data points, transactions can create a certain “velocity” to this data growth. As an example, the extraordinary growth of social media is generating new transactions and records. But the availability of ever-expanding data sets doesn’t guarantee success in the search for business value.
Data is now a factor of production
Data has become a full-fledged factor of production, like capital, labor and raw materials, and it’s not just a requirement for organizations with obscure applications in special industries. Companies in all sectors are combining and comparing more data sets in an effort to lower costs, improve quality, increase productivity and create new products. For example, analyzing data supplied directly from products in the field can help improve designs. Or a company may be able to get a jump on competitors through a deeper analysis of its customers’ behavior compared with that of a growing number of available market characteristics.
Storage must evolve
Big data has outgrown its own infrastructure and it’s driving the development of storage, networking and compute systems designed to handle these specific new challenges. Software requirements ultimately drive hardware functionality and, in this case, big data analytics are impacting the development of data storage infrastructures.
This could mean an opportunity for storage and IT infrastructure companies. As data sets continue to grow with both structured and unstructured data, and analysis of that data gets more diverse, current storage system designs will be less able to meet the needs of big data. Storage vendors have begun to respond with block- and file-based systems designed to accommodate many of these requirements. Here’s a listing of some of the characteristics big data storage infrastructures need to incorporate to meet the challenges presented by big data.
Capacity. “Big” often translates into petabytes of data, so big data storage systems certainly need to be able to scale. But they also need to scale easily, adding capacity in modules or arrays transparently to users, or at least without taking the system down. Scale-out storage is becoming a popular alternative for this use case. Scale-out’s clustered architecture features nodes of storage capacity with embedded processing power and connectivity that can grow seamlessly, avoiding the silos of storage that traditional systems can create.
Big data also means a large number of files. Managing the accumulation of metadata for file systems at this level can reduce scalability and impact performance, a situation that can be a problem for traditional NAS systems. Object-based storage architectures, on the other hand, can allow big data storage systems to expand file counts into the billions without suffering the overhead problems that traditional file systems encounter. Object-based storage systems can also scale geographically, enabling large infrastructures to be spread across multiple locations.
Latency. Big data may also have a real-time component, especially in use cases involving web transactions or finance. For example, tailoring web advertising to each user’s browsing history requires real-time analytics. Storage systems must be able grow to the aforementioned proportions while maintaining performance because latency can produce “stale data.” Here, too, scale-out architectures enable the cluster of storage nodes to increase in processing power and connectivity as they grow in capacity. Object-based storage systems can parallelize data streams, further improving throughput.
Many big data environments will need to provide high IOPS performance, such as those in high-performance computing (HPC) environments. Server virtualization will drive high IOPS requirements, just as it does in traditional IT environments. To meet these challenges, solid-state storage devices can be implemented in many different formats, from a simple server-based cache to all-flash-based scalable storage systems.
Access. As companies get better at understanding the potential of big data analysis, the need to compare differing data sets will bring more people into the data sharing loop. In the quest to create business value, firms are looking at more ways to cross-reference different data objects from various platforms. Storage infrastructures that include global file systems can help address this issue, as they allow multiple users on multiple hosts to access files from many different back-end storage systems in multiple locations.
Security. Financial data, medical information and government intelligence carry their own security standards and requirements. While these may not be different from what current IT managers must accommodate, big data analytics may need to cross-reference data that may not have been co-mingled in the past, which may create some new security considerations.
Cost. “Big” can also mean expensive. And at the scale many organizations are operating their big data environments, cost containment will be an imperative. This means more efficiency “within the box,” as well as less expensive components. Storage deduplication has already entered the primary storage market and, depending on the data types involved, could bring some value for big data storage systems. The ability to reduce capacity consumption on the back end, even by a few percentage points, can provide a significant return on investment as data sets grow. Thin provisioning, snapshots and clones may also provide some efficiencies depending on the data types involved.
Many big data storage systems will include an archive component, especially for those organizations dealing with historical trending or long-term retention requirements. Tape is still the most economical storage medium from a capacity/dollar standpoint, and archive systems that support multiterabyte cartridges are becoming the de facto standard in many of these environments.
What may have the biggest impact on cost containment is the use of commodity hardware. It’s clear that big data infrastructures won’t be able to rely on the big iron enterprises have traditionally turned to. Many of the first and largest big data users have developed their own “white-box” systems that leverage a commodity-oriented, cost-saving strategy. But more storage products are now coming out in the form of software that can be installed on existing systems or common, off-the-shelf hardware. In addition, many of these companies are selling their software technologies as commodity appliances or partnering with hardware manufacturers to produce similar offerings.
Persistence. Many big data applications involve regulatory compliance that dictates data be saved for years or decades. Medical information is often saved for the life of the patient. Financial information is typically saved for seven years. But big data users are also saving data longer because it’s part of an historical record or used for time-based analysis. This requirement for longevity means storage manufacturers need to include on-going integrity checks and other long-term reliability features, as well as address the need for data-in-place upgrades.
Flexibility. Because big data storage infrastructures usually get very large, care must be taken in their design so they can grow and evolve along with the analytics component of the mission. Data migration is essentially a thing of the past in the big data world, especially since data may be in multiple locations. A big data storage infrastructure is essentially fixed once you begin to fill it, so it must be able to accommodate different use cases and data scenarios as it evolves.
Application awareness. Some of the first big data implementations involved application-specific infrastructures, such as systems developed for government projects or the white-box systems invented by large Internet services companies. Application awareness is becoming more common in mainstream storage systems as a way to improve efficiency or performance, and it’s a technology that should apply to big data environments.
Smaller users. As a business requirement, big data will trickle down to organizations that are much smaller than what some storage infrastructure marketing departments may associate with big data analytics. It’s not only for the “lunatic fringe” or oddball use cases anymore, so storage vendors playing in the big data space would do well to provide smaller configurations while focusing on the cost requirements.
BIO: Eric Slack is a senior analyst at Storage Switzerland.
- How Australian organisations are coping with big data storage challenges –ComputerWeekly.com
- Beyond Big Data Storage for the AI/ML Era –Pavilion
Dig Deeper on Big data storage
On-premise object storage: Key products and use casesBy: Stephen Pritchard
Email firm runs millions of mailboxes with Scality object storage
Scality object storage scales to 400k mailboxes for NZ provider
Scale-out NAS: File storage with cloud and flash from big fiveBy: Stephen Pritchard