This article can also be found in the Premium Editorial Download "Storage magazine: CDP 2.0: Finding success with the latest continuous data protection tools."
Download it now to read this article plus other related content.
Just what is Ranger?
But Ranger isn't the first HPC cluster the university has deployed. TACC has three smaller clusters: two Dell Inc. Linux clusters--one with 5,840 compute processors and 176TB of disk storage, and another with 1,736 compute cores and 68TB of disk space--and an IBM Corp. Power5 System with 96 processors and 7.2TB of disk space. From its experience running these clusters, the university learned how to build the Ranger cluster.
"With Ranger, we knew it was going to be a very large system, so we tried to adopt what we had already learned on previous visits or experiences with small clusters," says Tommy Minyard, TACC's associate director, advanced computing systems.
For instance, Minyard and his team had experience with Linux and the open-source Lustre file system. "Based on our experiences of performance in a Linux-based environment over native InfiniBand, we decided Lustre was the appropriate decision," explains Minyard. "We have a lot of people on staff that have Linux and Lustre experience."
The Ranger cluster consists of 82 Sun Microsystems
| Inc. Blade 6048 Modular System racks, each populated with 48 Sun Blade server modules for a total of 3,936 compute nodes. Each blade has four quad-core AMD Inc. Barcelona processors for a total of 15,744 processors with 62,976 cores. These servers contain Mellanox Technologies Inc. ConnectX IB InfiniBand host channel adapters that connect to 24-port InfiniBand leaf switches and then to two redundant Sun Data- center Switch 3456, a 3,456-port InfiniBand switch.
The InfiniBand switches also connect to 72 Sun Fire X4500 (Thumper) servers that serve as I/O servers and in aggregate contain as much as 1.73 petabytes (PB) of SATA disk. Six meta data servers comprising Sun Fire X4600 M2 servers also connect to the InfiniBand switches; the meta data servers connect to a Sun StorageTek 6540 array with 9TB of storage and act as controllers for the Thumper object storage target arrays. All of this connectivity is managed and aggregated with the open-source Lustre file system and runs under CentOS, a Red Hat Enterprise Linux distribution.
The Ranger cluster features more than 570 petaflops; the Thumper storage servers, each with 24TB of disk, offer aggregate bandwidth of 72Gb/sec. The largest Lustre file system on Ranger offers 1PB of storage.
Work on Ranger is scheduled through the Sun Grid Engine. Ranger also uses Rocks provisioning software to handle operating system and app deployments, the OpenFabrics stack that controls the InfiniBand interconnect and two message passing interface (MPI) implementations: MVAPICH and Open MPI (see "Ranger cluster diagram," below).
This was first published in October 2008