Steve Prescott, director of technology at London-based Framestore, says the production house started with 50 TB a year ago for the project but had to add storage three times before completing work.
With 80% of the rendering needing to be done in three months to meet an Oct. 30 deadline so the film could open the weekend before Christmas, Framestore had 278 artists working on the movie. "The look we were going for was a Dutch Masters kind of thing, not shiny or cartoony," says Prescott. "Textures were important. It had to be perfect."
Prescott suspected from the start that he would have to add storage during the project, and wanted a system that would scale easily. "Sometimes," he says, "the way to get around a problem is to throw more resources at it."
For CGI rendering, Framestore used a supercomputer system made up of a 200 TB Linux cluster, mirrored the production data to a secondary cluster and installed a 10 Gbps dark fibre network to replicate its files across four London locations each day.
"We wanted to make sure [the storage system] was as resilient as possible," says Prescott. "We built a cluster behind the primary cluster to mirror and snapshot. If we lost the primary cluster, we had another cluster we could fall back to. If somebody deleted something, we could get it back straight away. With all that material, not having to put it onto slower storage or tape was quite important."
Lustre's unified naming system allows Prescott's artists to share files across different servers efficiently. "Before, we used lots of NFS servers and linked them," he says. "We had different servers for different parts of the project, and they'd be linked together in a Byzantine web. That became difficult to manage. The server would crawl, and rendering would go slowly."
Framestore's storage setup is more common in research laboratories or other supercomputing environments than in movie production. "We're probably the only people in our industry using it," says Prescott. "The standard model is to use it to compute a large problem. We have clusters running mixed applications, render farms with people animating on the cluster. We stress the system more than anybody."