Catching up with deduplication


This article can also be found in the Premium Editorial Download "Storage magazine: Surprise winner: BlueArc earns top NAS quality award honors."

Download it now to read this article plus other related content.

To avoid that scenario, ExaGrid Systems Inc.'s ExaGrid and Sepaton's S2100-ES2 execute only on backup sets that have completed, so deduplication doesn't impact backup and restore performance. On the first analysis of backed up data, ExaGrid and S2100-ES2 only compress the data and don't deduplicate. When a second backup completes, ExaGrid does byte-level delta differencing while Sepaton uses its ContentAware software to compare objects in the first backup at byte level against similar objects in the second one. Like objects in the first backup are then deleted and replaced with pointers to objects in the second backup, with objects in the second backup then compressed but not deduplicated. This deduplication and compression process repeats as backups occur.

How to estimate your deduplication ratio

The actual deduplication ratio--what you should expect to get and how soon you can get it--will vary according to many factors, some of which are within a user's control. Here are a few variables that can help you estimate the deduplication ratio you can reasonably expect to achieve.

Redundant data. The more redundant data you have on your servers, the higher the deduplication ratios you can expect to achieve. If you have primarily Windows servers with similar files and/or databases, you can reasonably

Requires Free Membership to View

expect to achieve higher ratios of deduplication. If your servers run multiple operating systems and different files and databases, expect lower deduplication ratios.

Rate of data change. Deduplication ratios are related to the number of changes occurring to the data. Each percentage increase in data change drops the ratio; the commonly cited 20:1 ratio is based on average data change rates of approximately 5%.

Precompressed data. Data compression is a key component in every vendor's data-reduction algorithm. Vendors base their advertised data-reduction ratios on the premise that compression will reduce already deduplicated data by a factor of 2:1. In a case where data deduplication achieves 15 times, compression could take that ratio up as high as 30:1. However, users with large amounts of data stored in compressed formats such as jpeg, mpeg or zip, aren't likely to realize the extra bump compression provides.

Data-retention period. The length of time data is retained affects the data-reduction rate. For example, to achieve a data-reduction ratio of 10 times to 30 times, you may need to retain and deduplicate a single data set over a period of 20 weeks. If you don't have the capacity to store data for that long, the data-reduction rate will be lower.

Frequency of full backups. Full backups give deduplication software a more comprehensive and granular view into the backup. The more frequently full backups occur, the higher the level of deduplication you'll achieve. Deduplicating backup software products have a slight edge over disk libraries because they run a full server scan every time they execute a server backup, even though they only back up changes to existing files or new files. In between full backups, disk libraries usually only receive the changes sent as part of the backup software's daily incrementals or differentials.

This was first published in June 2007

There are Comments. Add yours.

TIP: Want to include a code block in your comment? Use <pre> or <code> tags around the desired text. Ex: <code>insert code</code>

REGISTER or login:

Forgot Password?
By submitting you agree to receive email from TechTarget and its partners. If you reside outside of the United States, you consent to having your personal data transferred to and processed in the United States. Privacy
Sort by: OldestNewest

Forgot Password?

No problem! Submit your e-mail address below. We'll send you an email containing your password.

Your password has been sent to: