Forget 50% annual capacity growth. Shops with unstructured data regularly report data doubling every couple of months, fueling a run on high-capacity storage systems. But it isn't just about capacity. In many cases, the throughput of the system needs to scale as much as the data does.
At Movielink LLC, a joint venture among five major Hollywood studios that offers downloadable movies, it's quite possible storage capacity will double--maybe even triple--in the next six months, says Bruce Anderson, VP of Web engineering and operations at the Santa Monica, CA-based firm.
An Isilon Systems' IQ customer with two clusters totaling 9TB, Movielink is simultaneously bracing itself for new content deals that will expand the number of titles in its catalog, additional file formats, higher density encoding bit rates and, hopefully, a surge of post-holiday customers.
Movielink's scalability requirements have been satisfied by Isilon's OneFS clustered file system, which allows you to easily add capacity. "Adding another node is just a question of adding another node," says Anderson, referring to OneFS' ability to automatically recognize a new node, then resize and restripe the data accordingly. "It really is that easy."
But another concern of Movielink's is keeping up with customers' bandwidth needs. "We want to make sure that the network is the limiting factor; we don't want to starve the customer's last-mile connection," Anderson says. With individual download rates averaging 1.7Mb/sec, Movielink needs to prepare for peak weekend periods when tens of thousands of users may download movies simultaneously. Currently, the Isilon cluster offers between 2Gb/sec to 3Gb/sec of traffic, and adding additional nodes also increases throughput. "We've never seen the Isilon system tapped," Anderson says.
Aerial imaging firm North West Group, based in Calgary, Alberta, recently hit the 300TB mark of data on spinning disk. With nine specialized aircraft gathering data over North and South America, the firm adds data at a rate of approximately 4TB per day, says VP John Welter, and could thus double its data in just 75 days.
North West's data sits on a variety of Fibre Channel SAN arrays from various vendors, which it chooses based on market conditions. But the data on the arrays has a common file-serving front end--four Bobcat NAS gateways from ONStor.
The firm recently upgraded from Bobcat 2260s to new 2280 models, which feature the FastPath accelerator chip that increases streaming performance to 300MB/sec sustained reads and 265MB/sec sustained writes. The move was necessary because in the year since North West Group installed ONStor, the Linux cluster it uses to process its aerial data has increased from 32 nodes to 80, and the Bobcat gateways were having trouble feeding them with enough data. With the new 2280s, North West Group reports an additional 50% throughput. "ONStor only advertises 40% improvement," says Welter, "but because of our data--sequential, large block files--we're seeing even better performance than they are."