Storing hundreds of terabytes and even petabytes of data is no longer uncommon for organizations throughout a vast array of industries, but there's a big difference between big data sets composed of stagnant archived files and big data containing business-critical streaming media files. These are commonly referred to as big data at rest, and big data in motion.
When determining what type of storage should house a big data set, it's important to take into account how files will be used. While object storage and scale-out network-attached storage (NAS) are two of the most popular storage options for big data environments, media and entertainment companies that rely on data in motion shouldn’t overlook including higher-performance storage, such as solid-state drives (SSDs).
SearchStorage constructed this guide to help storage pros craft an architecture to house big data sets containing large streaming files. From the links provided throughout this guide, learn about the storage challenges you might encounter with big data and how to work around them, as well as the best way to perform analytics. You'll be well on your way to building the best architecture for your big data in motion.
Foundations of storing big data sets
Big data -- large amounts of unstructured data -- can yield positive business results when parsed effectively. But to get that competitive advantage, IT pros need to know how the technology works and what roadblocks they should anticipate in getting there. The obvious obstacle to storing big data is finding a platform that can house such a large amount of information, but what happens when large files, such as video and audio, require enough performance to stream? The following links provide insight on ways to deal with these problems, such as using compression or caching, and give tips on how storage type can make a difference.
Systems that provide fast access to metadata, caching and heavy compression help ease media-rich big data storage challenges, according to Wikibon Chief Technology Officer David Floyer. Continue Reading
Wikibon CTO David Floyer predicts object-based systems will be popular storage solutions for big data sets, and advises tape for long-form video files, disk for short clips. Continue Reading
In this Q&A, author Krish Krishnan shares his thoughts on the challenges of working with big data and the ways it has changed our approach to data. Continue Reading
Exploring big data storage options
Obvious storage choices for many big data platforms include scale-out NAS, for the amount of capacity it has, and object storage, which can help with the unstructured nature of big data sets. But when streaming files, it can also be a good idea to look at high-performance storage, such as SSDs. The stories and podcasts below include examples of what a storage system should provide for big data, and when specific types of storage work the best in big data environments.
Expert Jon Toigo discusses what big data is, the popularity of object storage and how to determine the best storage for a big data infrastructure. Continue Reading
In this podcast, Toigo describes how traditional storage infrastructures and practices have been changing to accommodate big data. Continue Reading
Read up on how big data technologies from companies such as GridIron, Quantum and RainStor address the challenges around retention, performance and capacity. Continue Reading
Data architects creating operational business intelligence applications may need to put streams of Hadoop data into a fast messaging infrastructure. Continue Reading
Choosing storage to accommodate big data analytics
Analytics is one of the most important parts of big data, but it can cause a lag in performance. In addition, the type of storage used can affect analytics efficiency. To make big data analytics effective, storage technologies, such as in-memory data grids, and advances in Hadoop continue to evolve. View the links below to learn more about how storage type affects analytics, and which tools can help you gain the most insight from the contents of big data sets.
New traits of in-memory data grids seem to target the needs of applications described as big data applications. Continue Reading
Cloud providers discuss the storage, networking and server challenges of big data analysis. Continue Reading
Big data storage: Experts weigh in
Still looking for more insights into big data and choosing the right storage for streaming files? Check out the videos below, in which expert contributors discuss the changing big data market and how storage plays an important role.
Storage analyst Ben Woo discusses why some storage options are better for big data than others, and what to consider when carrying out a big data project.
Consolidating data and virtualizing data are two central themes in any big data project, according to Ben Woo.
Big media will have an effect on just about every industry, according to James Kobielus, IBM's big data evangelist.
Progress Software's Jesse Davis says one big data trend is a move toward inquiry, insight and innovation.