Q
Evaluate Weigh the pros and cons of technologies, products and projects you are considering.

What are some downsides to using erasure codes with object storage?

Erasure coding is an effective tool for correcting errors in data systems, but when used with object storage systems, latency issues can arise.

Erasure coding is a forward error correction technology that's used to provide data resiliency and long-term data...

integrity, especially in large data systems. Erasure codes are often used instead of traditional RAID because of their ability to provide a more granular correction process, reducing the time and overhead required to reconstruct data (drive rebuilds).

Erasure codes (ECs) parse incoming data into multiple component blocks, then, somewhat like a parity calculation, expand each block with some additional information, creating a slightly redundant but more resilient superset of data. With a mathematical algorithm, the system can use these expanded blocks to recreate the original data set, even with missing or corrupted blocks. This allows the storage system to still deliver data, even after multiple drive or node failures.

Most erasure coding schemes allow the user to configure the level of resiliency, essentially by increasing the amount of parity data generated for each block. There are also different levels at which EC can be applied: at the array level, at the node level (for scale-out architectures) or at the system level -- which can affect how much processing overhead it consumes.

This is one of the possible downsides to using EC. It consumes compute resources, often significant amounts versus object replication, which simply copies the data x number of times. The overhead EC represents is dependent on where erasure codes are applied (at the array, the node or the system) and the level of resiliency chosen.

EC can be combined with data distribution or dispersion to improve resiliency and eliminate the need to make dedicated copies for off-site storage. This process essentially spreads data blocks across multiple nodes or systems, usually in different physical locations. However, using a distributed architecture where data blocks are spread between different physical locations can create a latency problem.

When distributed data must be reconstructed, network bandwidth quickly becomes the limiting factor as blocks are pulled across the WAN. Some object storage systems combine EC and replication, using erasure codes at the local system level and copying data between geographic locations to alleviate latency.

Next Steps

Using erasure coding can help reduce costs

The benefits of erasure coding do not come without some tradeoffs

This was last published in November 2014

Dig Deeper on Object storage

PRO+

Content

Find more PRO+ content and other member only offers, here.

Have a question for an expert?

Please add a title for your question

Get answers from a TechTarget expert on whatever's puzzling you.

You will be able to add details on the next page.

Start the conversation

Send me notifications when other members comment.

By submitting you agree to receive email from TechTarget and its partners. If you reside outside of the United States, you consent to having your personal data transferred to and processed in the United States. Privacy

Please create a username to comment.

-ADS BY GOOGLE

SearchSolidStateStorage

SearchCloudStorage

SearchDisasterRecovery

SearchDataBackup

Close