Amazon S3 API for cloud storage leads pack, for now

Vendors continue to add support for Amazon's S3 API, but will alternative APIs knock it down from its current status of the de facto standard?

One of the major hurdles to public cloud and object storage adoption has been cleared with the emergence of the Amazon Simple Storage application programming interface as a de facto standard.

Even if it proves to be a temporary standard, the Amazon Simple Storage (S3) application programming interface (API) gives developers a robust and simple way to write storage applications for the cloud. However, it hasn't ended the debate over what cloud storage API will ultimately win.

The lack of a standard had been cited as a problem for object storage, which serves as a building block of public clouds. Due to brand recognition and early user adoption of the Amazon cloud, storage vendors have chosen S3 as the API to write to instead of waiting for an industry standard.

Considering the popularity of the Amazon cloud, that's no surprise. However, not everyone is completely sold on the Amazon API as the permanent standard. Even some of the vendors whose support has made Amazon the de facto standard say they expect alternatives such as the Storage Networking Industry Association (SNIA)-backed Cloud Data Management Interface (CDMI) and OpenStack Swift to present a challenge.

"Almost everyone I talk to is writing to the Amazon S3 API," said Arun Taneja, founder, president and consulting analyst at Hopkinton, Mass.-based Taneja Group. "Amazon S3 has the largest storage cloud in the world, and if I were a software and hardware developer, the first [one] I would support is Amazon, then [Microsoft] Azure and Google. I would say that I'm not aware of any storage vendor that does not support Amazon on the back end. Any vendor with storage on the back end has no choice but to write to the Amazon API."

Object storage vendor Caringo Inc. recently added support for Amazon S3 in its CloudScaler 2.0 enterprise cloud application. CloudScaler joined object storage products such as Amplidata AmpliStor, Cleversafe, dsNet, EMC Atmos, Scality Ring and others that support the Amazon API.

"S3 has done a very good job of building an ecosystem of support of applications," said Adrian Herrera, Caringo's senior director of marketing. "We definitely see a demand for [the S3 API]. We see it in the enterprise and also see demand from the service provider side. Amazon has done a good job of getting customers comfortable with the cloud."

Scality CEO Jérôme Lecat conceded the Amazon S3 API is the de facto standard for REST-based cloud storage, although he is not sure that status is permanent.

"Many users experiment with S3, and at some point decide to build a dedicated cloud for themselves," he said. "Because they experimented with S3, the easiest way for them to deploy their dedicated cloud is with an S3 API."

What about CDMI?

Lecat said he still expects CDMI to gain wide adoption in the long run.

"Amazon has no program to support the use of the API by other cloud providers for storage vendors," he said. "CDMI is a far richer API. It's actually more comparable to a file system in terms of what you can do with it while still allowing distant access. So far, CDMI has had little traction, but I think CDMI will be used because there are use cases that cannot be solved with Amazon S3 that can be handled with CDMI. At the end of the day, I expect the Amazon S3 API and CDMI to coexist. I also expect other proprietary APIs to be used in special cases."

Taneja dismissed CDMI, however, because SNIA-based standards have a history of tepid market adoption.

"Who talks about SMI-S? No one," he said, referring to SNIA's Storage Management Initiative Specification.

Amazon S3 has so much customer adoption that "a lot of developers are writing to it, so it is the de facto standard," said Henry Baltazer, a senior analyst for infrastructure and operations professionals at Forrester Research Inc. in Cambridge, Mass.

The Amazon S3 API has 90 mashups, according to, up from 77 Amazon S3 mashups tallied last January. Introduced in 2006, Amazon S3 uses a simple Web services interface to store objects in the online Amazon infrastructure. The object has four parts -- metadata, value, key and access -- and is available in REST and SOAP protocols with responses formatted in XML.

Is S3 API fit for enterprise duty?

Scality's Lecat isn't alone among S3 API supporters who consider it lacking key enterprise features. The API doesn't have WORM, erasure coding, authentication or access control capabilities.

Caringo's Herrera said he considers it more of a checkbox item that customers look for when purchasing cloud-based products. He called it a simple API, more of a plug-and-play service. But that is one reason developers write backup, data management and content management applications to the API.

"We support the S3 API so these applications can write directly to Caringo storage," he said. "It's a pretty simple API. That is why I would not call it the de facto standard. It has a simple feature set that gives customers some control, like uploading and downloading, authentication and authorization. It's based on replication. It's a public cloud and they have no transparency. You send your data and have to take their word for it. You don't have access to the data center."

In comparison, Herrera said, Caringo's CAStor object storage gives customers full control of the data lifecycle management by customizing the metadata. CAStor also does replication and erasure coding, WORM, continuous integrity checks and has the ability to upgrade hashes that keep applications up-to-date with the latest algorithms.

"They have no data verification feature, meaning they can't verify the data has not been changed," Herrera said of Amazon. "That's a drawback for verticals that require regulatory services such as healthcare."

Taneja said the API is 100% robust because he has never heard anyone say, "I wrote to the Amazon API and we had errors." But for sophistication, he said he would give it a C.

Tom Leyden, director of product marketing at DataDirect Networks (DDN), said the open source OpenStack object storage API could be a solid competitor to the Amazon API. DDN supports the Amazon API as a separate layer to its Web Object Scaler (WOS) that is built on object storage. To use S3, you need a WOS cloud via an S3 gateway.

"We are seeing S3 used for file-sharing applications, not to support big projects," Leyden said. "We are seeing Swift in large projects, especially in companies with large engineering teams. We are seeing more demand for Swift, but we are leveraging the brand recognition for the S3 API."

Dig Deeper on Cloud storage management and standards