eBay keeps up with evolving storage demands


This article can also be found in the Premium Editorial Download "Storage magazine: Benefits of third-party data protection and recovery management tools."

Download it now to read this article plus other related content.

Few businesses have grown as rapidly as eBay. Its founder and chairman, Pierre Omidyar, built the online trading bazaar in his living room in one weekend in September 1995 with hardware that could be purchased at Fry's Electronics. Every sale item was a separate file, generated by a Perl script, and the system maxed out at 50,000 active items. Search functionality was nonexistent and storage was internal, directly attached.

Fast forward to 2007 and eBay now has 212 million users and 584 million listings. On a typical day, there are 1 billion page views and 26 billion SQL queries, which averages out to 6,000 SQL transactions per second, per database.

William Crosby-Lundin, eBay's SAN and high-availability infrastructure manager, says only 12 people manage eBay's storage, which consists of 13 discrete SANs that each offer a different quality of service for performance, availability and cost. Approximately 2 petabytes (PB) of raw storage (eBay adds 10TBs, 75 LUNs and performs six database moves each week) support 1,000 SAN-connected hosts and more than 600 database clusters.

To reach its 99.94% availability objective, eBay built a redundant path (sometimes as many as four per LUN) to each device in its SANs; it takes 8,000 ports to tie everything together. To increase performance and remove hot spots, Crosby-Lundin says there are multiple layers of aggregation and slicing of storage both at the host level and in the array. Ninety-five percent of its storage

Requires Free Membership to View

is devoted to databases, batch processing and file-transfer protocol operations, while 5% is NAS-based, used for email and fixed content. Crosby-Lundin won't name the brand of storage eBay uses--and for security reasons declined to talk about the company's disaster recovery policies and infrastructure--but did say the company keeps several copies of some databases, as many as four at multiple locations.

The complexity of eBay's storage environment requires heavy customization. "We wish we could, but we just can't buy all the stuff we need," says Crosby-Lundin, adding that in "most cases we have to write the code that makes the things we need happen." While "most standards-based tools deliver the lowest common denominator of functionality," he says, "we want more specific information for our relatively homogeneous storage environment." eBay uses a homegrown storage resource management tool to provision, monitor and manage its storage.

This was first published in July 2007

There are Comments. Add yours.

TIP: Want to include a code block in your comment? Use <pre> or <code> tags around the desired text. Ex: <code>insert code</code>

REGISTER or login:

Forgot Password?
By submitting you agree to receive email from TechTarget and its partners. If you reside outside of the United States, you consent to having your personal data transferred to and processed in the United States. Privacy
Sort by: OldestNewest

Forgot Password?

No problem! Submit your e-mail address below. We'll send you an email containing your password.

Your password has been sent to: