Evaluate Weigh the pros and cons of technologies, products and projects you are considering.

Does it matter if cloud providers use erasure codes or MCM?

Many admins may not care whether their cloud provider uses erasure coding or multicopy mirroring, but one approach has a bigger impact on protection.

Does it matter whether multi-copy mirroring or erasure codes are used by my cloud provider?

In my experience, most users don't care whether their public cloud provider uses multi-copying mirroring (MCM) or erasure codes -- but they should. The short answer is that erasure codes offer a better guarantee that your data will be protected. But since the price per gigabyte is approximately the same for companies that use MCM -- Google and Amazon, for example -- and offerings with erasure coding -- such as Microsoft's Azure, many customers ignore this debate when choosing a provider. However, MCM generally uses more storage, and is less resilient, than erasure codes.

With MCM, you are protected against only two concurrent losses. MCM is most effective when the mirrored object has some form of autonomic healing that periodically checks data object health and corrects it if there are any issues. If a disk drive or node has failed, gone dark, isn't there or comes back corrupted, the storage system will make another copy from a healthy version of the data objects and write them somewhere else. This system works pretty well, but it has a high cost associated with it because protecting against a single data object failure requires 100% additional storage capacity and nodes. Protecting against two concurrent data object failures requires 200% additional storage capacity and nodes; three concurrent failures require 300% and so on. It's an extremely high overhead.

With autonomic healing, erasure codes change that equation by protecting against six or more concurrent drive, node, system, site or other failures. It does this by dividing a data object into a number of chunks, the total being referred to as the width. A common width is 16 chunks. The object storage has to read only a subset of those chunks (referred to as the breadth) to reconstitute the data object. A common breadth is 10 chunks. In this example, the object storage system can tolerate six concurrent failures of drives, nodes, sites and so on, and still read the data objects. Autonomic healing enables the lost chunks to be recreated and written elsewhere. This example demonstrates three times the resilience of MCM at one-fifth the overhead. You therefore should care which data protection method is implemented by a cloud provider because it is more likely that data objects will be there when they're needed if erasure codes are used.

However, nothing is perfect. A downside to erasure codes is that additional latency can reduce performance. Depending on how many other latencies you are introducing, this can be an important factor to consider.

Dig Deeper on Public cloud storage

Join the conversation

1 comment

Send me notifications when other members comment.

Please create a username to comment.

Well, from a functional perspective you would use object replication for objects you need to read. For objects that you are unlikely to read use erasure coding to store them. In general, small objects should not be erasure coded. Yes, there is a performance penalty when reading erasure coded objects. An interesting example is Cloudian, which allows you to use erasure coding at the bucket level, so you can have some buckets that use erasure coding and others that use 3x object replication.