When it needed help mapping the sky, Fermilab chose a clustered file system
Every day, scientist Jim Annis receives about 12 DLT tapes full of data from a wide-field telescope in New Mexico. Annis and a team of astronomers are using the data to create a five-color, three-dimensional map of a quarter of the sky. The astronomy project is a relatively small effort for particle physics research center Fermilab, where Annis works, but it's a huge undertaking for the world of astronomy in general.
"What we'll have when we're finished will be a map of essentially everything you'd see if you went outside in March and looked up at the sky," says Annis. "This will be the first map of its kind."
Fermilab, situated on 6,800 acres just outside of Chicago, is best known for its primary mission -- running the world's highest-energy particle accelerator, the Tevatron. The lab employs about 1,000 scientists -- hundreds of whom work in the astronomy group.
Annis' group takes the raw data it gets from the daily shipment of data tapes via FedEx and analyzes it, a process that actually creates more data. The analyzed data is then stored and made available to Fermilab's collaborative astronomy group.
Annis is focusing on one particular segment of the project, in which he searches for clusters of galaxies in the catalog of data. He determines the brightness of galaxies, which helps measure the distance of the galaxies. "For this, I run a lot of compute nodes looking at essentially the same data," says Annis.
"I need fast access to lots of data -- maybe a terabyte of catalog data," says Annis. "And I can't currently do that by buying something from Dell and putting it on my desktop, because I need 10 or 20 compute nodes to have access to the data."
In the days of old, the group stored its data in about 10 Linux boxes with large IDE disk arrays. The system served data via NFS to scientists working on the project, all of whom needed frequent access. "It turned out that if I tried to hammer that data with 10 compute nodes, what usually happened was that the server would time out and the network would crash," Annis says. "All this did was annoy me."
Annis and his group looked at a number of options before deciding to switch their storage system to Sistina Software Inc.'s Global File System (GFS). GFS is a clustered file system that gives multiple servers on a storage area network (SAN) access to a single file system. The group members knew that they needed a shared file system and liked the fact that Sistina's GFS was open-source. They invested in a Fibre Channel switch and other hardware to create their SAN, and then installed GFS. "Now, all the compute nodes have really fast access to the same data, and I don't ever have things like server timeouts," says Annis.
Annis says that he and his group were attracted to GFS because it keeps the lock for files on the disk, so if a computer tries to ask the disk to access the file, the disk denies the request because someone else is using it. "To my mind, that's exactly where locks for files should live -- on the disk, so they know who's accessing them," says Annis. Most of the other vendors that the group spoke with had a lock server, one machine that would tell all the other machines whether they could access a file.
Since implementing the GFS system about two years ago, Annis says, his group has done a couple of upgrades. And each time, it's been a vast improvement. Also, within the past two years, the group has mapped out about 500 square degrees of sky. (For reference, the moon is about a quarter of a square degree.) When the project is complete, probably in 2007, the full map will cover about 10,000 square degrees.
"The most valuable resource to astronomers is their thinking time," says Annis. "If I have to think about my disks, I'm not a happy person. I want to just think about galaxies."
For more information on Fermilab, visit its Web site.
Additional information on Sistina software, click here.
More on this topic:
