Maximum SAN performance requires key components--host bus adapters and Fibre Channel directors--to work in harmony.
SANs are among the most mission-critical enterprise networks, so organizations tend to overbuy and overallocate SAN capacity to avert performance problems. Buying more capacity might work sometimes, but it isn't always an option. Storage administrators need to find ways to wring more value out of their SANs, and performance tuning is the place to start.
|SAN tuning checklist|
Use this checklist as a guide for performance tuning.
Host bus adapters (HBAs)
All SAN hardware vendors offer tools that configure, monitor and collect performance data for their devices. For example, Emulex Corp.'s HBAnyware allows administrators to discover host bus adapters (HBAs) and collect data about their status, attributes and performance. Brocade Communications Systems Inc.'s optional Advanced Performance Monitoring tool collects SilkWorm director performance data, which can be viewed in its Fabric Manager or exported to Microsoft Excel. EMC Corp.'s Navisphere Analyzer, an add-on to its Navisphere Management Suite, allows Clariion users to gather performance data and identify trends. Yet the key to SAN performance tuning is knowing what device settings to monitor--for HBAs, Fibre Channel (FC) directors, arrays or tape libraries--and when their parameters need adjusting (see "SAN tuning checklist").
Appropriately configuring an HBA before it's connected to the SAN is the best way to ensure that the HBA isn't the source of a SAN performance problem. Initial considerations when selecting and configuring an HBA include:
- Number of HBA ports
- Number of HBAs on the server
- Placement of HBAs on the server
- What load-balancing and path failover software to use
- Version and type of HBA driver and firmware
- Internal HBA settings
However, if a server supports VMware and multiple Windows partitions, it might make sense to deploy an HBA with four FC ports and have a port reserved for each Windows instance. It's generally recommended to deploy one high-performance application on a VMware server along with several lower performance applications; this way, the four-port card can be configured to give a lower performing application its own port with minimal or no impact to the higher performing application sharing the HBA.
The number of HBAs and where they're placed in the server can also affect performance. When using more than one HBA in a server, the HBAs must share the Peripheral Component Interconnect (PCI) bus and its allotted throughput.
"Sometimes the bus becomes a bottleneck," says Mike Smith, Emulex's executive vice president of worldwide marketing, limiting the HBA port's ability to transfer data between the host and storage. If multiple PCI devices share the same Peripheral Component Interconnect Extended (PCI-X) bus, the clock speed of each slot may be reduced, affecting the throughput capabilities of a particular slot. The telltale signs are a general slowdown in server applications and longer transaction latencies.
The use of path failover and load-balancing software should enhance server performance and availability. Software such as EMC's PowerPath and Symantec Corp.'s Dynamic Multipathing (DMP) identifies available HBA ports and then spreads I/O across the ports to increase the traffic flow between the server and the storage array. This software also helps to increase the server's availability. If an HBA fails, or one of the paths associated with it through the SAN becomes unavailable, the software labels the failed HBA as unavailable and reroutes traffic to other HBAs under its management. The main shortcoming of these products is that they generally work with only a limited number of storage arrays or require other foundational products, such as Symantec's Volume Manager.
You should also verify that the latest firmware and appropriate drivers are loaded on the HBAs before they're put into service. This ensures that the latest HBA-tuning features are installed and that the HBA is compatible with the various devices in the SAN environment. Most operating systems like AIX, Sun Solaris and Windows include HBA drivers that automatically detect and configure HBAs. But these drivers don't include optimized settings that some HBA vendors offer for certain vendor's storage arrays; they also don't alter the HBA's internal settings for performance-intensive applications.
Modern HBAs are automatically configured based on the number of target logical unit numbers (LUNs) they detect. For instance, the queue-depth setting--the number of pending I/Os an HBA can hold while waiting for responses from the target LUNs--is managed dynamically by the HBA's firmware. Initially set at 32, the queue-depth setting is adjusted as users add or remove LUNs.
But for high-performance environments, users need to configure the HBA's I/O coalescing setting. HBA vendors find that with most HBAs deployed in low-performance environments, this option is turned off even though each I/O generates a server CPU interrupt. Turning it on allows the HBA to pool I/O requests and bundle them. Once the bundle or buffer is full, only one server CPU interrupt will be generated for each bundle when it's sent, rather than producing one CPU interrupt for each I/O.
|Performance tips for arrays and tape drives|
Fibre Channel (FC) storage arrays
While configuring and tuning FC HBAs often occurs during the initial install, FC directors require more ongoing changes as the SAN evolves. Application, server and storage growth, backup configuration changes and connecting SAN islands creates a more dynamic environment for FC directors. Though FC directors usually provide ample bandwidth and throughput, performance may deteriorate as changes occur because:
- Multiple applications are accessing the same storage port
- A zone includes too many server ports
- Server placement is on edge and core SAN switches
Resolving this issue requires access to the director zoning information and possibly a performance-measurement tool. The director management software will identify all of the HBAs zoned to a specific storage port and which port they're plugged into on the FC SAN. With the ports identified, you can determine which application is the culprit and take corrective action.
If an informal review doesn't immediately reveal the offending application, each FC director vendor offers tools that capture and monitor information from specific FC ports. Brocade's SAN Health, Fabric Watch, Fabric Manager and Advanced Performance Monitoring tools can help identify the problem. SAN Health is a free download from Brocade's Web site and can be used on any vendor's FC director to take a performance sample. Brocade's Fabric Watch product can be configured to send an alarm if a port exceeds a certain performance threshold, while its Fabric Manager tool provides historical performance data. The Advanced Performance Monitor tool provides more advanced diagnostics on how FC traffic moves through the fabric and how the SAN is designed; it only needs to be used if the other tools can't pinpoint the performance problem.
|When to use an SRM tool|
Storage resource management (SRM) tools such as Hewlett-Packard Co.'s Storage Essentials and EMC Corp.'s ControlCenter offer optional features that collect and correlate performance data from SAN devices. In some cases, the tool can manage the devices or even dynamically tune them based on pre-set policies. Yet when compared to the free or low-cost tools that ship with storage devices, it's sometimes difficult to justify their purchase. Here are some tips to help you decide if and when deploying an SRM tool is warranted:
In a large environment with multiple SAN devices, the hassle of learning and managing each vendor-specific tool makes a comprehensive SRM application that's capable of tuning all of the devices a cost-effective alternative.
Check to see if the SRM tool can act as a dashboard that launches device-specific management tools.
The biggest benefit of an SRM tool is its ability to visualize, correlate and analyze performance data from all of the SAN devices that the application data touches or passes through.
SRM tools generally operate under the assumption that one IT group has control of all the devices in a SAN--from the host down to the storage array. This may not be the way your shop operates.
While SRM vendors provide flexible product licensing, you'll still have to negotiate. Consider deals that tie SRM licensing to the total number of managed SAN ports or total amount of managed storage capacity--those are the numbers that tend to outstrip server and CPU growth over time.
Too many server ports in the same zone can also degrade SAN performance. This will most likely occur during server boot ups, and while probing the SAN for new storage ports or existing storage ports for new LUNs. When the server initiates these probes, the server HBA starts to probe each port it has access to on the SAN. Although the server HBA software should recognize the difference between server HBAs ("initiators") and storage and tape HBAs ("targets"), some HBA drivers are problematic and probe server HBAs until they timeout, delaying server boots and new volume discoveries. The best way to avoid this is to keep each server HBA in its own zone with its own set of targets.
How and where application servers are attached to a growing SAN can also affect performance. Administrators may be tempted to simply attach new servers to a new FC switch or director, connect the new switch to an existing SAN director using an inter-switch link (ISL) and allocate storage. This is a bad idea for two reasons. First, new directors will generally have higher performance characteristics, and storage and tape devices generally should have first dibs on the highest performing ports. Second, the technology on new servers may support higher levels of SAN performance, such as 2Gb/sec or 4Gb/sec speeds, and if forced to access storage on the core SAN director via an ISL, performance will degrade to the level of the slowest component in the storage path.
With Cisco Systems Inc. SANs, the MDS 9000's SAN-OS remote switched port analyzer (RSPAN) functionality can identify these types of performance bottlenecks. The RSPAN utility mirrors traffic from one or more FC interfaces to another port in the fabric, and monitors traffic without the insertion of a probe. The traffic may also be captured and encapsulated by Cisco's optional Port Analyzer Adapter and sent to a PC for SCSI I/O and FC frame-level analysis.
The best way to avoid these performance problems is to keep the highest performing technology at the core. This typically involves moving the highest performing application servers, storage arrays, virtual tape libraries and tape drives to the new core director and relocating lower performing applications to edge switches. When allocating core director ports, priority should be given to the highest performing applications, such as backup, databases and file servers.
An ongoing process
Of course, SAN tuning is a work in progress. You need to constantly monitor and tinker with your storage fabric (see "Performance tips for arrays and tape drives," and "When to use an SRM tool") on a regular basis to get the best performance. By taking simple steps to tune SAN components before they're installed in the SAN, and knowing which ones to analyze first when performance hiccups occur, the number of performance issues will be greatly reduced down the road.