This article can also be found in the Premium Editorial Download "Storage magazine: Storage managers reveal details on tape reliability."
Download it now to read this article plus other related content.
|Options for scaling Fibre Channel storage networks|
Storage interfaces are typically configured to be undersubscribed to avoid congestion that could hamper performance. Non-blocking architectures provide full bandwidth between port pairs to reduce the possibility of congestion. But not all servers have the same requirements and performance needs, so their workloads may be aggregated to a single port on a storage device with faster ports. Access to the switch bandwidth may be shared by two or more adjacent ports instead of all ports operating at line or interface speed.
Oversubscribed ports lower costs while dispelling the myth that all servers should be treated with the same interface. With its MDS 9500 switch, Cisco has implemented oversubscribed host-optimized ports; the box's standard 16-port 2Gb FC blades operate at full speed, but its 32-port blades are oversubscribed by design to share internal bandwidth across multiple ports. Cisco has also implemented host-optimized ports on its 9000 series of edge switches.
The caveat related to oversubscribing ports is that the placement and locality of servers and storage becomes important to prevent blockage and congestion. For example, if all servers need to operate at full speed, the oversubscription ratio would be 3.2:1. This means that, assuming 100% utilization, there could be up to 3.2Gb/sec of server and storage I/O workload competing for 1Gb/sec of available switching bandwidth. For example, if you have four 2Gb/sec oversubscribed ports that fan-in to a 2.5Gb/sec interface in a switch core, assuming 100% utilization, this would result in an oversubscribed ratio of 3.2:1. In reality, many servers have relatively low I/O and workload requirements that, with planning, can be accommodated without performance delays on an oversubscribed port. But caution is required when oversubscribing ports. It's important to keep track of which ports are for low-performance servers and which ones are for the high performers that may need more bandwidth.
A configuration where many servers converge to a single storage port is called fan-in (or fan-out if viewed from the storage out to the servers). In a fan-in topology, servers and storage devices could be on full-speed or oversubscribed ports. Congestion may occur with a fan-in architecture at the port because of insufficient bandwidth or head-of-line blocking. Head-of-line blocking takes place when traffic at the beginning of the queue blocks other traffic.
Virtual output queues can help to solve the head-of-line blocking problem. A virtual output queue essentially creates a logical port for traffic from various servers while having shared access to the physical port resources. Virtual output queue functionally has become a standard capability on current-generation switching products and is sometimes referred to as a quality of service feature. This eliminates head-of-line blocking by providing fair access and preventing delays caused by slower devices accessing the port. In this manner, virtual output queues can help to scale performance and connectivity.
ISLs and horizontal scaling
ISLs are a key component for implementing horizontal scaling on a local or wide-area basis. One way to increase bandwidth and availability among switches is to add more ISLs aggregated using trunking or configured as individual ISLs. Another method is a hybrid approach using a combination of faster ISLs and additional ISLs to meet specific availability and performance needs. The number of ISLs needed between switches is a function of the amount of bandwidth that applications require and the port speed of the ISL. For example, a 7Gb/sec bandwidth required between two switches could be accommodated with seven 1Gb ISLs, four 2Gb ISLs, two 4Gb ISLs or one 10Gb ISL. Additional ISLs may be configured for load balancing and redundancy. The level of performance needed in a failure situation (such as the loss of an ISL, switch or communication circuit) will determine how to distribute the ISLs among the switches.
FC switches can typically be physically stacked on top of each other and interconnected with ISLs. Some switch vendors, such as QLogic Corp., provide dedicated 10Gb ISL ports. A stackable switch is a good scaling option in a rack or cabinet where additional ports may be added over time, rather than using a larger, frame-based director switch or connecting traditional switches with ISLs.
A virtual ISL is a new scaling technique that allows ports in different logical domains in the same physical switch to communicate. Virtual ISLs are also known as zero-cost or zero-overhead ISLs, and are available on newer switching products that use an internal backplane rather than an external physical ISL (ports and cables).
This was first published in February 2005