Essential Guide

Get started Bring yourself up to speed with our introductory content.

Choosing storage for streaming large files in big data sets

Big data in motion requires storage that can handle streaming and analytics efficiently. Learn how to avoid setbacks and build an effective infrastructure with this guide.


Storing hundreds of terabytes and even petabytes of data is no longer uncommon for organizations throughout a vast array of industries, but there's a big difference between big data sets composed of stagnant archived files and big data containing business-critical streaming media files. These are commonly referred to as big data at rest, and big data in motion.

When determining what type of storage should house a big data set, it's important to take into account how files will be used. While object storage and scale-out network-attached storage (NAS) are two of the most popular storage options for big data environments, media and entertainment companies that rely on data in motion shouldn’t overlook including higher-performance storage, such as solid-state drives (SSDs).

SearchStorage constructed this guide to help storage pros craft an architecture to house big data sets containing large streaming files. From the links provided throughout this guide, learn about the storage challenges you might encounter with big data and how to work around them, as well as the best way to perform analytics. You'll be well on your way to building the best architecture for your big data in motion.


Foundations of storing big data sets

Big data -- large amounts of unstructured data -- can yield positive business results when parsed effectively. But to get that competitive advantage, IT pros need to know how the technology works and what roadblocks they should anticipate in getting there. The obvious obstacle to storing big data is finding a platform that can house such a large amount of information, but what happens when large files, such as video and audio, require enough performance to stream? The following links provide insight on ways to deal with these problems, such as using compression or caching, and give tips on how storage type can make a difference.


The truth about big data technology

Big data technology is a big deal for storage shops, and a clear understanding of what it means is required to configure storage for big data applications. Continue Reading


Big data problems? Metadata, caching, compression could help

Systems that provide fast access to metadata, caching and heavy compression help ease media-rich big data storage challenges, according to Wikibon Chief Technology Officer David Floyer. Continue Reading


How the media and entertainment industry influences big data storage

Wikibon CTO David Floyer predicts object-based systems will be popular storage solutions for big data sets, and advises tape for long-form video files, disk for short clips. Continue Reading


Management and storage challenges presented by big data sets

In this Q&A, author Krish Krishnan shares his thoughts on the challenges of working with big data and the ways it has changed our approach to data. Continue Reading


Tips for streaming files of big data sets

Solid-state drives are a good choice for streaming large video files. Other factors to consider include cost and how often video will be accessed. Continue Reading


Dealing with big data files in real time

There's a real difference between big data at rest and big data in motion. To get big data moving, complex event processing and other technologies may be needed. Continue Reading

2Storage options-

Exploring big data storage options

Obvious storage choices for many big data platforms include scale-out NAS, for the amount of capacity it has, and object storage, which can help with the unstructured nature of big data sets. But when streaming files, it can also be a good idea to look at high-performance storage, such as SSDs. The stories and podcasts below include examples of what a storage system should provide for big data, and when specific types of storage work the best in big data environments.


Storage for big data infrastructures: Types and considerations

Expert Jon Toigo discusses what big data is, the popularity of object storage and how to determine the best storage for a big data infrastructure. Continue Reading


How storage options for big data sets are changing

In this podcast, Toigo describes how traditional storage infrastructures and practices have been changing to accommodate big data. Continue Reading


Classifying big data storage options

We explore distributed nodes, scale-out NAS, all-SSD arrays and object storage as possible choices for a storage architecture for big data sets. Continue Reading


Storage technologies address big data requirements

Read up on how big data technologies from companies such as GridIron, Quantum and RainStor address the challenges around retention, performance and capacity. Continue Reading


Big data in motion vs. big data at rest: Creating an effective architecture

Data architects creating operational business intelligence applications may need to put streams of Hadoop data into a fast messaging infrastructure. Continue Reading


Choosing storage to accommodate big data analytics

Analytics is one of the most important parts of big data, but it can cause a lag in performance. In addition, the type of storage used can affect analytics efficiency. To make big data analytics effective, storage technologies, such as in-memory data grids, and advances in Hadoop continue to evolve. View the links below to learn more about how storage type affects analytics, and which tools can help you gain the most insight from the contents of big data sets.


Effect of big data storage type on analytics

There are two basic types of big data analytics -- synchronous and asynchronous -- but they both have big data storage appetites and specialized needs. Continue Reading


Processing big data without the performance lag

Low-cost, solid-state memory is powering high-speed analytics of big data streaming from social network feeds and the industrial Internet. Continue Reading


Why big data analytics is worth the hassle

Companies of all sizes can benefit from performing analytics on big data sets. Learn the role various technologies might play in a big data environment. Continue Reading


Technologies evolve to make big data analytics easier

New traits of in-memory data grids seem to target the needs of applications described as big data applications. Continue Reading


Avoid the challenges of big data analytics in the cloud

Cloud providers discuss the storage, networking and server challenges of big data analysis. Continue Reading


Using storage to suppress bottlenecks with Hadoop

A variety of performance issues can bog down Hadoop clusters. However, there are ways to sidestep the pitfalls and keep your big data environment humming. Continue Reading


Big data storage: Experts weigh in

Still looking for more insights into big data and choosing the right storage for streaming files? Check out the videos below, in which expert contributors discuss the changing big data market and how storage plays an important role.


Storage options for big data sets

Storage analyst Ben Woo discusses why some storage options are better for big data than others, and what to consider when carrying out a big data project.


Using scale-out options for storing big data

Consolidating data and virtualizing data are two central themes in any big data project, according to Ben Woo.


Big data transforms into 'big media'

Big media will have an effect on just about every industry, according to James Kobielus, IBM's big data evangelist.


Big data moves past problems

Progress Software's Jesse Davis says one big data trend is a move toward inquiry, insight and innovation.

Start the conversation

Send me notifications when other members comment.

Please create a username to comment.