Feature

Compression puts the squeeze on dedupe

Ezine

This article can also be found in the Premium Editorial Download "Storage magazine: CDP 2.0: Finding success with the latest continuous data protection tools."

Download it now to read this article plus other related content.

DEDUPLICATION IS ALL the rage these days, while the good old-fashioned Lempel-Ziv (LZ) compression technology we all grew up with is taken for granted. The real differences between dedupe and compression have to do with algorithms or how each one does its job. But not many of us need to know algorithm differentials to do our jobs.

"I think there's a lot of confusion in the marketplace," says Arun Taneja, founder and consulting analyst at Taneja Group in Hopkinton, MA. "Most people aren't computer scientists."

Both technologies shrink data volumes. Compression does it by squeezing out repetitive bits in a data stream, cleaning redundant data within a file. Dedupe compares objects at the file and sub-file level, removes duplicate files by referencing the original and saves only one instance.

Standard compression technology provides about a 2:1 ratio, halving the number of bits in a stream. Both tape and disk can accommodate compression. Dedupe, which is strictly for disks (tape can't accommodate its serial nature), boasts ratios on the order of 30:1. "But it's not like one is a panacea and one is a dud," notes Taneja.

Now for the twist: Storwize and Ocarina Networks, two competing compression vendors, have rejiggered the LZ algorithm so that compression technology can be used in "those scenarios where performance matters," says Taneja,

Requires Free Membership to View

not just for backup data. Storwize calls itself a capacity optimization provider for primary storage. That means Storwize offers inline compression (that's radical) and promises no performance drawbacks.

Amit Bar-On, IT manager at Polycom Inc., a Pleasanton, CA-based video conferencing firm with 3,000 users, connected Storwize to his NetApp system. Right away, his 300GB of available storage jumped to 1.1TB. He says Polycom's data load was reduced by 65% across multiple apps.

Dedupe wasn't an option he needed, says Bar-On, although NetApp offers the capability for free in its system. "Dedupe is searching for files, and if you're doing that in the middle of the day, it can influence performance," he says. "I needed to compress all the data, not search the duplicate files, and I wanted something to run in real-time."


--Ellen O'Brien

This was first published in October 2008

There are Comments. Add yours.

 
TIP: Want to include a code block in your comment? Use <pre> or <code> tags around the desired text. Ex: <code>insert code</code>

REGISTER or login:

Forgot Password?
By submitting you agree to receive email from TechTarget and its partners. If you reside outside of the United States, you consent to having your personal data transferred to and processed in the United States. Privacy
Sort by: OldestNewest

Forgot Password?

No problem! Submit your e-mail address below. We'll send you an email containing your password.

Your password has been sent to: