Denver-based Photobucket has accumulated 17 petabytes of data since 2003. That prompted the image- and video-hosting website to add a second data center earlier this year to ensure it could scale storage and provide reliable uptime to its 100 million
The company stores more than 13 billion images and claims 5 million more are uploaded to the site each day. "One of the challenges we've always had is just the sheer number of files we have because we store multiple copies of each image -- the original, a Web-friendly copy and a thumbnail," said Michael Clark, Photobucket's chief technology officer.
Photobucket had been using NetApp's Data Ontap since the early days of the company, Clark explained, but decided to use the move to its new Phoenix data center as the reason to look at its storage options. "We decided it was the perfect time to reevaluate," he said. "We just wanted to see if there was somebody doing NFS better [and] somebody doing data replication better and more cost-effectively."
The company's Denver and Phoenix data centers are almost identical, with NetApp clustered Data Ontap and Data Ontap 8 implemented at each site, along with NetApp FAS3210, 3240, 6040 and 6240 arrays. The storage supports 1,100 servers and 500 virtual servers, and the environment is based on a FlexPod architecture using a KVM hypervisor.
The company also uses an in-house object storage system for its TinyPic.com domain that stores almost 2 billion media objects.
Before purchasing NetApp's clustered Data Ontap, Photobucket tested systems from Hitachi Data Systems and several commercial object storage platforms. But after the IT team ran into performance problems with the object storage options and compared overall data center costs, Photobucket stuck with NetApp.
Clark said that while Photobucket couldn't directly migrate data from older volumes to the clustered Ontap and had to retrain the IT team because of changes to the command-line interface, purchasing the clustered Data Ontap product was worth it after the initial implementation.
Among the most notable benefits from the clustered version is nondisruptive maintenance. Clark said that helped his team make it through a recent disk failure without downtime or data loss. "All we did was migrate the data, split the volumes to a couple of other controllers, did all the work we needed to do, and brought it back up and over," he said. "We were able to do that nondisruptively and really quickly. It surprised me how easy and simple that worked out of the box."
Clark said NetApp's SnapMirror feature was very useful when it came to backing up large amounts of data. "As we brought in other storage systems to test, one of the challenges that we ran into, especially in the enterprise storage platforms, is [that] none of the replication technologies were practical for that number of files," he said.
With SnapMirror, Photobucket can snapshot all data to FAS2240 and FAS3210 arrays every 30 minutes and maintain up to a month of snapshots at all times.
In addition to continued data migration to the clustered Ontap product, Photobucket is rolling out a new object storage infrastructure.