Do you like cruising Google Earth? It might not be the same without a storage area network (SAN) file system from Advanced Digital Information Corp. (ADIC).
One of its main suppliers, DigitalGlobe Inc., which collects
DigitalGlobe, which owns QuickBird, the world's highest resolution commercial satellite -- it boasts a 2-foot pixel capable of zooming in to the windshields on cars -- is one of the principal suppliers of satellite images to Google Earth, the National Geospatial-Intelligence Agency and the U.S. Geological Survey.
The company was formerly managing its near petabyte of storage on a collection of Silicon Graphics Inc., Sun Microsystems Inc., Linux and Windows servers, local RAID arrays and Scalar tape libraries from ADIC. To send images between the servers so that different photo-editing applications could process them, the company had to post each file to an FTP server.
"We were limited to 100 Mbps at a time," Childs said. For a company adding 150 gigabytes of data per day from its satellite, that wasn't going to cut it.
Last year, the company finally consolidated its online storage with a 105 terabyte (TB) Sun 9990 array, otherwise known as Hitachi Data Systems Inc.'s Tagmastore. The SAN would give them the speed and data protection of block-level storage, but in order to preserve its file-based production process, DigitalGlobe layered on ADIC's StorNext SAN file system software.
Childs said he chose ADIC primarily because of its scalability. "Most proprietary file systems won't go over 2 TB. This one goes to over 100 TB," he said.
Moreover, according to Childs, the ADIC system works efficiently. "We used to have to clean off our file server nightly. We'd get calls at 2 a.m. to come and unclog it, basically," he said. "The StorNext can handle 90 days' worth of files and automatically deletes them after that time period."
During the evaluation process several years ago, DigitalGlobe looked at "five or six different vendors," Childs said, but declined to name them. When it came time to implement the file system last year, StorNext was the only one at the time that would support all the different operating systems DigitalGlobe was using.
"The others would do three out of four, but this was the only one that had them all covered," Childs said.
Childs also said that among the proprietary file systems, he liked the failover strategy of StorNext the best -- a power device forces the acting metadata server to power off if it's corrupted, so it doesn't keep trying to take over the file system.
"It's still not great," Childs said. For example, StorNext's reporting capabilities left something to be desired.
"What's most difficult right now is actually understanding what's going on behind the file system," he said. "We'd like better debug analysis for systems admins -- there are some command line tools but nothing very in-depth. There's some home-grown analysis which we've done but it's hard to get a moment-to-moment glance at what's happening with the whole system."
Ultimately, the addition of the SAN and the file system have turned business around for DigitalGlobe. Where a single process job used to take 10 to 12 hours, it now takes less than three.<p>
"Our volumes are up drastically. The overall pixels we're getting out the door this year are up 80% this year," Childs said.
One big difference is in dealing with server maintenance, he said. "It used to be an incredibly manual process to restart a server in the midst of a job. We'd have to start completely over and remember where the image was in the process stream and reload it."
Now, the ADIC file system "freezes" the data where it is while hardware issues are dealt with and restores them automatically.
"It saves us hours," Childs said.
The time saved by the SAN and its file system is being put to good use -- DigitalGlobe will soon launch another satellite and grow its U.S. government business. When that happens, Childs said, the storage volume for the firm will expand by three or four times.
But, according to Childs, preliminary performance tests show the StorNext file system can handle 20 times more throughput.
"We think storage isn't something we need to worry about for the next five to 10 years," he said.
This was first published in February 2006