News Stay informed about the latest enterprise technology news and product updates.

StorReduce offers inline deduplication for cloud object stores

Startup StorReduce ventures into uncharted territory with an inline deduplication tool for reducing cloud object stores in Amazon S3.

Storage startup StorReduce has emerged from stealth with software that provides inline deduplication of petabyte-scale cloud object stores in Amazon Web Services' Simple Storage Service.

StorReduce formally incorporated in the U.S. in November after completing a proof of concept with its first customer, data management firm SpectrumData. The software, also known as StorReduce, gets installed atop a hierarchical storage management layer that uses a Simple Storage Service (S3) connection.

StorReduce instances are available on the Amazon Marketplace. The virtual appliance uses cloud storage for persistent data, running either as an Amazon Elastic Compute Cloud instance or on physical servers that migrate on-premises object stores to a private cloud.

Block-level inline dedupe to S3 instance

StorReduce sits between client applications and public cloud storage to deduplicate unstructured data inline in a constant stream at the block level and pass only the unique blocks through to an S3 instance. The software then hashes the changes and stores an index layer in the cloud for data hydration.

Because the software doesn't buffer data, StorReduce claims inline deduplication of objects can be achieved at sustained speeds up to 600 Mbps. Getting that throughput rate requires customers to have a local cache of solid-state drives (SSDs) to maintain index data.

The company said a single StorReduce server with SSD local storage can handle up to 10 PB of raw data. StorReduce suggests 5 TB of local SSDs for every petabyte of object storage.

Dedupe for cloud 'more interesting' than SMB backup

StorReduce technology originated as part of Triple Backup, an Australian backup vendor that served small and medium-sized businesses (SMBs). Triple Backup created virtual images of storage servers and sent the images to cloud storage via a low-bandwidth connection.

"To move images to the cloud, it was necessary for Triple Backup to develop a deduplication engine. We eventually ported the engine onto the cloud, and that's when we realized there didn't seem to be anyone else doing backend inline deduplication for cloud storage," said CEO Vanessa Wilson, who cofounded StorReduce with fellow Triple Backup colleagues CTO Hugh Emberson and chief product officer Mark Cox.

The trio shifted gears from SMB backup, launching StorReduce and relocating to Sunnyvale, Calif., to refine the software for cloud-based object stores. Inline deduplication for the cloud "seemed to be a much more interesting market to us than SMB backup," Wilson said.

As part of its proof of concept with StorReduce, SpectrumData migrated tens of thousands of legacy backup tapes to virtual tapes in Amazon Web Services.

The test involved slightly reconfiguring StorReduce software to run as a local instance rather than in the Amazon cloud.

SpectrumData director Guy Holmes said StorReduce software provided two major benefits.

"First, we were able to output one-tenth the data that we would have previously. Second, we got 10 times the data throughput. Whereas Amazon was going to charge me one dollar for that chunk of storage, I'm now paying 10 cents by using StorReduce," Holmes said.

Emberson, the StorReduce CTO, said his company plans to release an object interface for OpenStack Swift cloud storage in June and anticipates a similar rollout for Microsoft Azure in 2016.

Next Steps

Using inline deduplication for an overall smaller data footprint

Deciding between object and block storage for your cloud

public cloud storage: What data is safe to move?

Dig Deeper on Cloud object storage