Storage for high-performance computing

Ezine

This article can also be found in the Premium Editorial Download "Storage magazine: CDP 2.0: Finding success with the latest continuous data protection tools."

Download it now to read this article plus other related content.

Just what is Ranger?
UT Austin installed the Ranger cluster in late 2007 as the result of a $59 million grant from the National Science Foundation. Of that figure, $39 million was used to buy the hardware and software that makes up the cluster, while $20 million was allocated for four years of managing and operating the cluster.

But Ranger isn't the first HPC cluster the university has deployed. TACC has three smaller clusters: two Dell Inc. Linux clusters--one with 5,840 compute processors and 176TB of disk storage, and another with 1,736 compute cores and 68TB of disk space--and an IBM Corp. Power5 System with 96 processors and 7.2TB of disk space. From its experience running these clusters, the university learned how to build the Ranger cluster.

"With Ranger, we knew it was going to be a very large system, so we tried to adopt what we had already learned on previous visits or experiences with small clusters," says Tommy Minyard, TACC's associate director, advanced computing systems.

For instance, Minyard and his team had experience with Linux and the open-source Lustre file system. "Based on our experiences of performance in a Linux-based environment over native InfiniBand, we decided Lustre was the appropriate decision," explains Minyard. "We have a lot of people on staff that have Linux and Lustre experience."

The Ranger cluster consists of 82 Sun Microsystems

    Requires Free Membership to View

    Login

Inc. Blade 6048 Modular System racks, each populated with 48 Sun Blade server modules for a total of 3,936 compute nodes. Each blade has four quad-core AMD Inc. Barcelona processors for a total of 15,744 processors with 62,976 cores. These servers contain Mellanox Technologies Inc. ConnectX IB InfiniBand host channel adapters that connect to 24-port InfiniBand leaf switches and then to two redundant Sun Data- center Switch 3456, a 3,456-port InfiniBand switch.

The InfiniBand switches also connect to 72 Sun Fire X4500 (Thumper) servers that serve as I/O servers and in aggregate contain as much as 1.73 petabytes (PB) of SATA disk. Six meta data servers comprising Sun Fire X4600 M2 servers also connect to the InfiniBand switches; the meta data servers connect to a Sun StorageTek 6540 array with 9TB of storage and act as controllers for the Thumper object storage target arrays. All of this connectivity is managed and aggregated with the open-source Lustre file system and runs under CentOS, a Red Hat Enterprise Linux distribution.

The Ranger cluster features more than 570 petaflops; the Thumper storage servers, each with 24TB of disk, offer aggregate bandwidth of 72Gb/sec. The largest Lustre file system on Ranger offers 1PB of storage.

Work on Ranger is scheduled through the Sun Grid Engine. Ranger also uses Rocks provisioning software to handle operating system and app deployments, the OpenFabrics stack that controls the InfiniBand interconnect and two message passing interface (MPI) implementations: MVAPICH and Open MPI (see "Ranger cluster diagram," below).


Ranger cluster diagram
The configuration of the Ranger cluster relies heavily on storage--72 Sun Fire X4500 Servers (Thumper) connect to the InfiniBand fabric and then to the 3,936 compute nodes in the cluster. Six meta data nodes running under the Lustre file system manage and organize the data stored on the Thumper servers. The only Fibre Channel in the system is the 32-port switch that connects the meta data nodes with their shared storage.


This was first published in October 2008

Join the conversationComment

Share
Comments

    Results

    Contribute to the conversation

    All fields are required. Comments will appear at the bottom of the article.
    Back to top