Object storage systems have been promoted as "the next big thing" for nearly a decade, though they have yet to...
develop a sufficiently compelling rationale to displace conventional file system-based storage in most environments. However, as storage becomes increasingly populated with large and small files -- so-called "unstructured data" -- wrangling all of this data in a hierarchical structure of volumes, directories and subdirectories supported by complex structures of nodes and inode extensions is becoming more and more unwieldy.
In some cases, businesses report storing hundreds of millions of very small files in a single directory used to build a webpage. In other cases, firms have digitally recorded medical images or surveillance video as files that need not only to be searched for and retrieved quickly, but modified with the addition of highly granular metadata to enable their efficient use. In short, the file system is beginning to fray at the edges, as storage requirements change.
Object storage involves the replacement of traditional file system structures with an object-oriented framework. Basically, instead of organizing files in a hierarchical file tree, data objects and files are organized into containers or "buckets," each with their own unique ID or key. Access is made simply via key or ID. Most metadata about the object or file is stored with the object or file, reducing the metadata management burden found in most file systems.
What exactly is object storage? It is a storage architecture that addresses and handles individual units of storage. It then assigns the objects a unique identifier, so the objects can be retrieved without knowing their physical location.
Basic object storage systems are simple to implement and operate, and rather minimalist in terms of functionality. Users can store, retrieve, copy and delete objects, and specify which users have what permissions to perform which functions. Increasingly, World Wide Web Consortium REST standards are being paired with object storage to facilitate integration with other applications and traditional file system-based data access methods. RESTful APIs and primitives are used to interact with containers and objects, enabling the use of external search engines -- in the case of popular public cloud object stores -- and other application software functions to interact directly with objects -- in the case of some video editing systems.
Object storage systems also let users define the metadata stored with the object to a much greater and much more granular extent than is possible with most file systems. This is very good news for archival applications, big data analytics and in other cases where data volumes are large and detailed metadata searching can yield great value.
Object storage is also thought to make better use of disk capacity than a traditional file system, since space is not reserved for incomplete files and because journals of directory paths and file names typically consume little space on disk. In fact, in the more elegant implementations of object storage systems -- such as Caringo Swarm -- only objects are written to disk, while a small stateless journal is maintained in RAM, which considerably speeds up operations.
With control over object structure and metadata, better use can be made of data protection services. Data copy operation can be applied to objects that actually require multiple separate copies based on metadata-centric policies. Files that are less frequently changing may be delegated to processes such as erasure coding rather than mirroring.
Some vendors are working to find ways to organize better data in caches and buffers by recoding it initially as an object, then using object metadata to assist in cache management. Amazon has implemented object storage in its Simple Storage Service and supports it with specialty REST-based protocols that have become as much a part of the storage interconnect lexicon as Fibre Channel, iSCSI or NFS.
Remember, if you are storing data to cloud services, chances are you are already using object storage systems today. If you aren't, it’s time to consider how object storage can benefit you.
The differences between object, file and block storage
Evaluating open-source object storage
Alternatives to object storage systems in the cloud
Many object storage use cases point back to the cloud
Dig Deeper on Object storage
Related Q&A from Jon Toigo
Linear Tape File System and Linear Tape-Open technology can improve user access and durability in your tape archive system. Explore specific products... Continue Reading
Parallel computing technology has not seen widespread use in the business world, but could that change? Jon Toigo discusses parallel I/O for ... Continue Reading
Software-defined storage architecture can be implemented in several different forms that all expose software functionality to hardware across an ... Continue Reading
Have a question for an expert?
Please add a title for your question
Get answers from a TechTarget expert on whatever's puzzling you.