Ezine

This article can also be found in the Premium Editorial Download "Storage magazine: Expanding SANs: How to scale today's storage networks."

Download it now to read this article plus other related content.

Four years ago, when business unit managers at the Royal Bank of Canada (RBC) Financial Group in Toronto began to understand the advantages of networked storage, storage area network (SAN) deployments at the $15.5-billion multinational financial services company gained momentum faster than Lance Armstrong accelerating down the Alps.

Putting its new gear to the test

    Requires Free Membership to View

In 1999, officials at Royal Bank of Canada (RBC) learned the hard way that compatibility testing is essential. The company set out to deploy its first storage area network (SAN), a tape SAN. Without first checking its tape vendor's compatibility matrix, RBC chose to use what Harold Durnford, the manager of systems-managed storage, now calls an "off-brand host bus adapter." But when Durnford's team attempted to put the shiny new SAN into production, it started experiencing throughput problems and frequent crashes.

"We'd have a failure every couple of days," says Durnford. "It got to the point that we'd have scopes and tracers running constantly. The only solution turned out to be to change the actual host bus adapter (HBA). Later, we discovered the device we'd chosen wasn't on the tape vendor's compatibility list. From then on, we tried to make sure someone else had tested the equipment first and that we did our own tests as well."

RBC set up its own SAN compatibility testing lab at a 70,000-square-foot data center in Ontario, approximately 90 miles northwest of the company's main data center in Toronto. The facility, with between 30TB and 40TB of storage at any given time, also serves as RBC's disaster recovery site. That enables SAN equipment testers at the site to get access to a complete set of the Windows and Unix servers configured identically to production servers. Storage devices and HBAs for testing are usually provided on a 90-day trial basis by vendors.

While no data center, the testing lab consumes the equivalent of three full-time employees, says Durnford. Currently, the lab is testing new StorageTek 9940B tape drives.

The first step in compatibility testing at RBC, according to Durnford, is to check equipment vendors' own compatibility testing results. RBC officials have learned to expect trouble when vendors say equipment "should" be compatible. But even when compatibility is guaranteed, RBC tests just to be sure.

The company has a standard menu of tests. They include:

  • End-to-end connection: Can the single host see all the disk? Are the sizes correct? Can it see the other host disk?
  • Cluster/failover: Does a clustering configuration (AIX or Sun, for example) see all of its own and other servers' disk?
  • Multipathing: Are all paths accessible?
  • HBA firmware and drivers: Can HBA and firmware at least connect and coexist?
  • Throughput: Do the HBAs deliver the rated throughput--1Gb switch or 2Gb switch--end-to-end?
  • Shared ports: What hosts (HBAs and drivers) can still function when sharing storage ports on a switch?
  • Software and firmware upgrades: Does all of the above still work after software is upgraded?
Testing has revealed that the most persistent problems have cropped up in multipathing software, says Durnford, particularly when RBC has attempted to run storage devices from more than one vendor on the same server. Often, multipathing software used by the different storage devices conflicts. In fact, testing has shown that it's better simply not to mix storage on a server, says Durnford.

RBC also has learned to retest whenever changes in firmware or software occur. For example, Durnford's team retested each cluster when, earlier this year, it upgraded from 1Gb to 2Gb switches.

Until then, rbc banks, brokerage business, insurance companies and other units had been saddled with a growing mess of direct-attached disk systems running on a wide range of servers--from Novell NetWare to IBM AS/400s. As the number of direct-attached drives--many of them IBM 7133 SSA serial disk systems--increased, RBC's far-flung businesses were having more and more trouble keeping up. When an application required more disk, provisioning took up to eight weeks. And because of the limits on SCSI bus cable lengths, many data centers were simply running out of room for more direct-attached storage (DAS).

So when RBC's central technical support division gave SANs the green light after having tested SAN firmware interoperability and issuing best practice recommendations, RBC's business units began switching to Dell, EMC and IBM SANs as soon as leases on older servers expired.

"We'd tell them, 'By the way, when your leases expire, you're ordering your next server with Fibre Channel cards. This is the firmware, the brand and literally, the part number to order.' And they did it," says Harold Durnford, RBC's manager of systems-managed storage who oversaw much of the switch to SANs.

SAN island proliferation
RBC's businesses complied with Durnford's recommendations perhaps just a bit too enthusiastically. It wasn't too long before SAN islands were popping up all over the place. And with there was no central IT authority to dictate when, where and under what conditions a new SAN should be deployed, RBC then found itself recreating many of its pre-SAN management problems, only this time it was because of SAN proliferation.

While RBC managed to restrict its Unix environment to two 256-port dual fabric SANs, its Windows environment was a different story. Rapidly growing and highly distributed applications such as Exchange e-mail were increasing total Windows-related storage by upwards of 100% per year. By early 2003, Durnford says, RBC found itself with upwards of 30 Windows-based SANs--many using 8- or 16-port gigabit switches--spread throughout the 30 countries in which RBC operates.

"On the Windows side, the proliferation of islands came about because the solution we were using would only provide for two terabytes per SAN," says Durnford. "Each application server might want one tenth of that, so after we got 10 servers using the SAN, with server No. 11, I'd have to start another SAN island." The result was that RBC was hiring more people to manage SAN islands and taking even longer to provision storage, says Durnford.

Now RBC is moving to end--or at least slow down--the SAN island proliferation. Durnford has put in place what amounts to a two-phase plan. The first plan for RBC is to use inter-switched links (ISLs) and larger, 2Gb switches to consolidate existing Windows SAN islands, while continuing to manage SANs supporting the company's different operating environments--Windows, Unix, mainframe and IBM iSeries--separately. In Phase 2, the company plans to use a common storage resource management (SRM) software tool to manage all of its SANs and may even build centralized storage utilities that remote servers could access using LAN-free ISLs running over dark fiber connections.

"Our initial exploitation of SANs was purely connectivity and volume purchase-driven," says Durnford, adding: "Now we've got to consolidate onto fewer SANs that we can view and manage more simply."

RBC is certainly not alone in attempting to stem SAN island proliferation by consolidating onto fewer, larger fabrics. According to estimates by Gartner Inc., SANs with a port count of 320 or greater will grow at a rate of 40% over the five-year period ending in 2007. That compares to a growth rate of 12% for SANs of all sizes over the same period.

Simplify storage provisioning
Like RBC, many organizations consolidating storage onto larger SANs are hoping they can simplify storage provisioning.

"Next to backup, provisioning is the number one issue that IT is dealing with. At most companies, it's in the dark ages," says Arun Taneja, founder and consulting analyst of the Taneja Group in Hopkinton, MA. "Usually, you've got the systems administrator, database administrator and the storage administrator trying to figure out why an application isn't running fast enough. By the time they figure out they need more storage and go through the process of buying and deploying it, what should have taken a few minutes, takes weeks. The move from direct-attached storage to SANs was supposed to fix that, and it has made things easier," says Taneja, adding, "but not easy enough."

In Phase 1 of RBC's SAN consolidation push, a first step was to establish standard SAN deployment, management and procurement processes and encourage the company's lines of business to use them.

Eighteen months ago, Durnford's group created a storage steering committee, which included IT and business representatives from each of the company's businesses.

The purpose of the committee, says Durnford, was to announce SAN management and consolidation best practices, such as when you're consolidating SANs; how and in what sequence you hook up the switches; how you document the changes and how you back up data during the change and restore it. "We wanted to make sure we shared our experiences so that they could financially benefit from them, avoiding costly mistakes," Durnford says.

And authority over storage still remained with the lines of business. But with business executives and IT managers on the steering committee, Durnford was then able to generate business buy-in for storage consolidation.

Interoperability testing
Among the key best practices that Durnford's group emphasized was the need to thoroughly test and verify the interoperability of SAN elements, including switches, directors, arrays, etc.--before putting consolidated SANs into production. When in 1999, the technical support division started evaluating SANs, Durnford's group tested interoperability of various devices at different release levels from different vendors. They soon learned to anticipate compatibility problems, says Durnford, even in cases where vendors had certified interoperability between their devices. In a consolidated SAN, interoperability testing is particularly important.

"It doesn't matter what the vendor tells you, oftentimes when you hook up two switches that are not of agreeable [release] levels, you'd get what providers like to call 'unexpected results,'" says Durnford. "In fact, oftentimes when we heard from the vendor that it should work, it was a clue to us that we needed to test it, certify it and lock down the firmware."

Centralizing information about what devices and firmware releases actually work with each other and disseminating it to the rest of the organization, says Durnford, accelerates SAN consolidation. The technical support division also developed standard change management procedures that should be used when deploying and consolidating SANs.

Just-in-time purchasing
As part of Phase 1 of its plan, RBC standardized its SAN hardware vendors and then centralized storage procurement with Durnford's group. On the Windows side, for example, RBC will use Dell Computer Corp., EMC and IBM gear. On the Unix side, it will be EMC and IBM. Boiling down vendors to just this group will greatly reduce compatibility problems as more SAN consolidation projects take place. And centralizing procurement allows RBC not only to make sure all groups are buying only from the approved list, but it will also allow for much speedier provisioning.

RBC has begun what Durnford calls "just-in-time" provisioning. What this means is that vendors ship switches and disk arrays to RBC just before they are really needed. Durnford's group then configures them and tests them in advance. Only once a SAN has reached an agree-upon threshold of consumed storage space is the additional disk deployed. And only then does the company actually pay for it. Using this approach, Durnford says, RBC has been able to cut the time it takes to provision new SAN storage from up to eight weeks to, in some cases, in only 24 hours. (For a look at a different SAN consolidation project also intended to simplify provisioning, see "AXA Group's major consolidation push")

Using these procedures and standards, RBC businesses have begun to consolidate their existing SANs. In one case, six older Windows-based SANs in three different sites--totaling 11TB--are being consolidated into three SANs. In another, multiple SANs supporting 50 Exchange servers deployed across two sites are being consolidated along with the application servers.

RBC is also consolidating the mainframe side, mainly just to reduce software license charges. Last year, the company consolidated the North Carolina data center supporting its United States-based Centura banking business to an existing Toronto data center. Another consolidation project is under way, this time RBC's West Coast data center is merging with its Toronto site.

For now, says Durnford, RBC plans to continue consolidating SANs within each operating environment without creating central SANs that can support servers running different operating systems. Currently, says Durnford, the company is shooting for creating standard SAN fabrics of 500 ports. Although there are few technical reasons to avoid creating consolidated heterogeneous SANs, RBC is wise to focus first on consolidating within operating environments, says Taneja.

"There is LUN masking software for switches and HBAs that allow you to consolidate storage for different operating systems onto one SAN fabric, but organizations should probably think twice before attempting that," says Taneja.

This was first published in November 2003

There are Comments. Add yours.

 
TIP: Want to include a code block in your comment? Use <pre> or <code> tags around the desired text. Ex: <code>insert code</code>

REGISTER or login:

Forgot Password?
By submitting you agree to receive email from TechTarget and its partners. If you reside outside of the United States, you consent to having your personal data transferred to and processed in the United States. Privacy
Sort by: OldestNewest

Forgot Password?

No problem! Submit your e-mail address below. We'll send you an email containing your password.

Your password has been sent to: