Erasure coding is a forward error correction technology that's used to provide data resiliency and long-term data...
integrity, especially in large data systems. Erasure codes are often used instead of traditional RAID because of their ability to provide a more granular correction process, reducing the time and overhead required to reconstruct data (drive rebuilds).
Erasure codes (ECs) parse incoming data into multiple component blocks, then, somewhat like a parity calculation, expand each block with some additional information, creating a slightly redundant but more resilient superset of data. With a mathematical algorithm, the system can use these expanded blocks to recreate the original data set, even with missing or corrupted blocks. This allows the storage system to still deliver data, even after multiple drive or node failures.
Most erasure coding schemes allow the user to configure the level of resiliency, essentially by increasing the amount of parity data generated for each block. There are also different levels at which EC can be applied: at the array level, at the node level (for scale-out architectures) or at the system level -- which can affect how much processing overhead it consumes.
This is one of the possible downsides to using EC. It consumes compute resources, often significant amounts versus object replication, which simply copies the data x number of times. The overhead EC represents is dependent on where erasure codes are applied (at the array, the node or the system) and the level of resiliency chosen.
EC can be combined with data distribution or dispersion to improve resiliency and eliminate the need to make dedicated copies for off-site storage. This process essentially spreads data blocks across multiple nodes or systems, usually in different physical locations. However, using a distributed architecture where data blocks are spread between different physical locations can create a latency problem.
When distributed data must be reconstructed, network bandwidth quickly becomes the limiting factor as blocks are pulled across the WAN. Some object storage systems combine EC and replication, using erasure codes at the local system level and copying data between geographic locations to alleviate latency.
Using erasure coding can help reduce costs
The benefits of erasure coding do not come without some tradeoffs
Dig Deeper on Data management tools
Related Q&A from Eric Slack
With increasingly demanding workloads, NAS systems can have a hard time keeping up. Can using NAS with an object storage option relieve some of these...continue reading
While scale-out storage has its benefits, object storage is becoming a more popular option. Learn about cloud object storage advantages in this ...continue reading
Eric Slack of Storage Switzerland discusses whether it is necessary to back up data stored in file sync services in this Expert Response.continue reading
Have a question for an expert?
Please add a title for your question
Get answers from a TechTarget expert on whatever's puzzling you.