How to design a core/edge SAN

The SPICE formula offers a creative and easy way to determine the best design for an efficient core/edge SAN.

SPICE makes it easy to compare several vendor configurations. For the sake of comparison, assume that a PI of seven will be achieved.

The Brocade 24000 core switch has 128 ports. Assume you chose the Brocade 3900 edge switch with 32 ports as the best choice for your SAN. This configuration would support 448 servers and 64 storage ports. There would be 16 edge switches. C = ports on core switch = 128 E = ports on edge switch = 32 ( C / 2 ) * PI = "S" or server capacity of core switch ( 128 / 2 ) * 7 = 448 servers ( C / 2 ) = number of storage ports ( 128 / 2 ) = 64...

storage ports Roundup ( S + ( S / PI ) ) / E = number of edge switches needed Roundup ( 448 + ( 448 / 7 ) ) / 32 = 16 edge switches



The Cisco MDS 9509 core switch has a variety of configurations, making it more complex to consider. Nevertheless, in a standard core/edge configuration, only the 16-port blades will be used in each of the seven slots available. The 32-port blades couldn't be used to connect to storage devices if all storage devices are simultaneously used at full capacity. Nor are they useful for the ISLs, which are designed to match the bandwidth available to the storage ports. In this configuration, the MDS 9509 has 112 ports on seven blades with 16 ports each. To complement this core switch, a Cisco 9140 with 40 ports will support 392 servers attached to 11 edge switches. This configuration supports 56 storage ports.
C = ports on core switch = 112
E = ports on edge switch = 40
( C / 2 ) * PI = "S" or server capacity of core switch
( 112 / 2 ) * 7 = 392 servers
( C / 2 ) = number of storage ports
( 112 / 2 ) = 56 storage ports
Roundup ( S + ( S / PI ) ) / E = number of edge switches needed
Roundup ( 392 + ( 392 / 7 ) ) / 40 = 12 edge switches



The McData 6140 core switch has 140 ports. Assume you chose the McData 4500 with 24 ports as the best value to meet your availability needs. The total number of servers a single core switch would support would be 490. To achieve this, it would be complemented with 24 24-port McData 4500s. It would have 70 storage ports attached to the core switch.
C = ports on core switch = 140
E = ports on edge switch = 24
( C / 2 ) * PI = "S" or server capacity of core switch
( 140 / 2 ) * 7 = 490 servers
( C / 2 ) = number of storage ports
( 140 / 2 ) = 70 storage ports
Roundup ( S + ( S / PI ) ) / E = number of edge switches needed
Roundup ( 490 + ( 490 / 7 ) ) / 24 = 24 edge switches



Three storage area network (SAN) architectures have become prominent: island, collocated and core/edge. Each topology serves a particular niche, but of the three, the core/edge SAN is the most scalable and widely deployed. Designing a large core/edge SAN can be a complicated process, but the SPICE algorithm greatly reduces the complexity.

The S variable is the number of servers that will be migrated initially to the core/edge SAN. It's the milestone by which the implementation project is measured. As with any project, server and capacity requirements may change during the implementation, but setting the S goal early in the planning phase will establish a clear completion point for the project.

Because a goal of the core/edge SAN is to keep the storage port as close to full utilization as possible, the P in the SPICE equation will be equal to I (and will be referred to as the "PI"). For example, if it takes 10 servers to keep a storage port nearly full, then the same 10 servers that fill the pipe between the disk and core switch will also fill the equivalent of a pipe between the core switch and an edge switch, namely an interswitch link (ISL).

The C and E variables are largely determined by the switch's port count. The C variable will be a determining factor in the total server capacity of the core/edge topology, but the E variable isn't important. If, for example, the PI that works in your environment is seven, then seven servers will share one ISL. If you purchased 16-port edge switches, you would need three of them to support 42 servers; if you purchased 24-port switches, you would need only two. Either selection results in a capacity of 42 servers and six ISLs leading to the core switch and six storage ports.

The most challenging task in designing the core/edge SAN topology is getting the PI right. It will vary, depending on applications and platforms. Large Unix Oracle servers tend to have a lower PI than small Wintel file servers. In a SAN with a widely diverse collection of servers and applications, the overall PI might need to be broken down into the various groups that make up the SAN environment. Although it may take some effort to determine an accurate PI for your SAN, there are significant rewards for doing so.

This SPICE algorithm can be applied to a variety of vendors' hardware, so it's possible to determine the bill of materials from each vendor to compare prices. You may even use your existing SAN to determine the correct SPICE for your new core/edge SAN. 

SPICE: Core/edge SAN math

What's the total number of servers a core switch will support?

Answer: (C/2)* PI
How many ISLs will an edge switch need?

Answer: Roundup (E/(PI + 1))
What's the total number of servers an edge switch will support?

Answer: E - Roundup (E/(PI + 1 ))
What's the total number of edge switches a core switch will support?

Answer: (C/2)/(E/(PI + 1))
What is the total number of storage ports a core switch will support?

Answer: (C/2)
If you have 28 servers that fully utilize four storage ports, how many core ports will you need?

Answer: C = (2 * PI) = (2 * 4) = 8
If you have 28 servers that fully utilize four storage ports, how many edge switches will you need?

Answer: Edge switches = S/(E - (E/(PI + 1))) = 2

Core/edge topology
In contrast to a SAN island or collocation SAN, the core/edge SAN offers the highest degree of availability for the most critical SAN component: the disk frame. While a server may be highly critical to the application manager, it's not nearly as critical as the disk frame that's shared among many servers. Therefore, one precept of the core/edge topology is that all storage devices should be located on a highly available directorclass or core switch. The servers can be located on the low-priced ports of the edge switches.

The disk frame is usually the most expensive SAN component and, under ideal conditions, it should be fully utilized before a new frame is purchased. A core/edge SAN design goal is to devise an architecture that allows the disk frame to become the bottleneck in the SAN, rather than the network between the frame and its contingent of servers. To achieve this objective, the bandwidth between the disk and core switch should be highly utilized. If there is a 2Gb/sec pipe between a disk port and the core switch, servers can be added until their average utilization approaches that 2Gb/sec limit. The number of servers required to reach this limit is usually referred to as the fan-out ratio of disk to servers.

The building block of the core/edge SAN is the core switch, which is populated with ISL and storage connections. The ISL connects the core switch to the edge switch attached to the servers.

In a core/edge SAN, the bandwidth needed by average servers is easily determined as the bandwidth of a pipe, usually 2Gb/sec, divided by the PI (the number of servers per storage port or the number of servers per ISL). If the PI has been accurately determined and a particular server becomes a bandwidth hog, then only that server needs to be moved. Because the architecture is designed to handle the clearly defined average performance of a server on an edge switch, the "hog" server would need to be migrated onto the core switch or the architecture is put at risk. When the PI is clearly defined, then the process for dealing with anomalies is easier to address; servers can be shuffled around to meet their capacity needs.

It's better to underestimate the PI than overestimate it. For example, a PI of seven means that two 24-port edge switches could support only 42 servers with six ISLs. If the PI had been estimated at 11, then those same two edge switches would support 44 servers with only four ISLs. If a PI of 11 was used, but a PI of seven was more accurate, then the ISLs between the edge and core would become the bottleneck and the disk could never be fully utilized. Conversely, if a PI of seven had been used, there would be excess capacity that could be used after the successful implementation of the SAN. Most importantly, lowering the PI to a safely attainable goal results in a core/edge design that contains room for additional hardware; raising it too high will result in an underfunded project.

The PI also determines overall capacity of the SAN (see "Applying the PI to Brocade, Cisco and McData switches"). For any configuration of core and edge switches, the PI sets how many servers one core switch can support. After that limit is reached, a new core switch is required. Regardless of size, the core switch will be roughly evenly divided between ports dedicated to storage devices and ISLconnections (see "SPICE: Core/edge SAN math"). Some core ports will need to be reserved for servers with higher bandwidth needs.

If you're migrating from an island or collocation SAN topology into a core/edge design, you can gauge the PI requirements of the core/edge SAN by gathering metrics from the SAN servers on the legacy environment. The PI gathered from the legacy SAN will be lower than what can be achieved once the storage devices are consolidated and can be more widely shared.

Once you have determined the PI that reflects your applications and their storage usage, you can quickly set a number of topology parameters, such as determining how many edge switches you'll need for your core/edge SAN, or how many servers a core switch can support.

This was first published in November 2004

Dig deeper on SAN management



Enjoy the benefits of Pro+ membership, learn more and join.



Forgot Password?

No problem! Submit your e-mail address below. We'll send you an email containing your password.

Your password has been sent to: