If the three guiding principles in real estate are 'location, location, location,', the guiding principles in designing a SAN are 'locality, locality, locality.' Because high locality produces significant cost savings and performance improvements, it is worth setting up a SAN to maximize locality.
Locality refers to localizing SAN traffic on a single switch or in one area of a SAN. It is not the same as topology, which refers to how the servers, switches and storage devices are connected together. Instead locality measures how the traffic flows over the SAN's topology.
The guiding principle of locality is that as much as possible all the storage traffic between a server or a group of servers should flow through only one switch. Obviously this requirement is automatically met in simple SANs having only one switch. Things get more complicated, and potentially a lot less efficient, when the SAN has a fabric of several switches.
Non-locality costs in multi-switch fabrics because each Inter-Switch Link (ISL) between switches takes at least two ports (one on each switch) and ports are usually the critical resource in a SAN fabric. Since each device on one switch that communicates with a device on the other takes an ISL, a traffic pattern that has four devices communicating across switches takes four ISLs. If locality isn't considered in laying out the SAN's traffic patterns, much of each switch's capacity can be eaten up by ISLs.
Locality considerations become more important as the complexity of the SAN grows. Finding the best arrangement of servers and storage is fairly easy in a two-switch fabric, but it becomes more difficult as the number of switches grows to ten or 20 or more. A core-edge topology can ease the problem by providing central switches or directors at the core of the fabric to switch between devices connected to switches as the edge of the fabric.
The key to maximizing locality is analyzing traffic patterns across the SAN. The SAN design team should consider carefully how to group the storage devices and servers to keep them in the as logically close as possible, even though they may be separated by physical distance. If the SAN is replacing systems with locally allocated storage the relation between existing servers and storage devices can provide a pattern for connecting them via SAN. In general you want to keep the storage now connected directly to a server on the same switch as the server in the SAN. In the event of a completely new system it is usually easy to allocate storage to servers on the same switches or in the same part of the SAN. This gets trickier when the design becomes more complex, as when the SAN incorporates existing devices as well as a substantial number of new ones. Often setting up a spreadsheet will help the designers visualize the relationships between the servers and storage and aid in keeping as much traffic local as possible.
Locality isn't just a matter of SAN design. SAN administration plays a vital role in maintaining locality by allocating storage and servers in such a way as to keep them on the same switch or in the same area as much as possible. This is a balancing act that tends to change over time as storage requirements grow and more servers are added. The need to maintain locality becomes another factor to consider in allocating storage or adding servers. It can act as a constraint on other factors, such as maximizing storage utilization. If the under-utilized storage device is on a different switch that the servers needing more storage, it may actually make more sense to accept the under-utilization and add a storage array to the switch with the servers than to try to use the existing, under-utilized storage and have to add more ISLs, and possibly additional switches, to carry the load.
Brocade Communication Systems Inc. discusses locality and its importance in a white paper titled "Locality In SAN Design", which is available at the company's Web site at: www.brocade.com.
Rick Cook has been writing about mass storage since the days when the term meant an 80K floppy disk. The computers he learned on used ferrite cores and magnetic drums. For the last twenty years he has been a freelance writer specializing in storage and other computer issues
