Too many SAN islands

Until then, rbc banks, brokerage business, insurance companies and other units had been saddled with a growing mess of direct-attached disk systems running on a wide range of servers--from Novell NetWare to IBM AS/400s. As the number of direct-attached drives--many of them IBM 7133 SSA serial disk systems--increased, RBC's far-flung businesses were having more and more trouble keeping up. When an application required more disk, provisioning took up to eight weeks. And because of the limits on SCSI bus cable lengths, many data centers were simply running out of room for more direct-attached storage (DAS).

So when RBC's central technical support division gave SANs the green light after having tested SAN firmware interoperability and issuing best practice recommendations, RBC's business units began switching to Dell, EMC and IBM SANs as soon as leases on older servers expired.

"We'd tell them, 'By the way, when your leases expire, you're ordering your next server with Fibre Channel cards. This is the firmware, the brand and literally, the part number to order.' And they did it," says Harold Durnford, RBC's manager of systems-managed storage who oversaw much of the switch to SANs.

SAN island proliferation
RBC's businesses complied with Durnford's recommendations perhaps just a bit too enthusiastically. It wasn't too long before SAN islands were popping up all over the place. And with there was no central IT authority to dictate when, where and under what conditions a new SAN should be deployed, RBC then found itself recreating many of its pre-SAN management problems, only this time it was because of SAN proliferation.

While RBC managed to restrict its Unix environment to two 256-port dual fabric SANs, its Windows environment was a different story. Rapidly growing and highly distributed applications such as Exchange e-mail were increasing total Windows-related storage by upwards of 100% per year. By early 2003, Durnford says, RBC found itself with upwards of 30 Windows-based SANs--many using 8- or 16-port gigabit switches--spread throughout the 30 countries in which RBC operates.

"On the Windows side, the proliferation of islands came about because the solution we were using would only provide for two terabytes per SAN," says Durnford. "Each application server might want one tenth of that, so after we got 10 servers using the SAN, with server No. 11, I'd have to start another SAN island." The result was that RBC was hiring more people to manage SAN islands and taking even longer to provision storage, says Durnford.

Now RBC is moving to end--or at least slow down--the SAN island proliferation. Durnford has put in place what amounts to a two-phase plan. The first plan for RBC is to use inter-switched links (ISLs) and larger, 2Gb switches to consolidate existing Windows SAN islands, while continuing to manage SANs supporting the company's different operating environments--Windows, Unix, mainframe and IBM iSeries--separately. In Phase 2, the company plans to use a common storage resource management (SRM) software tool to manage all of its SANs and may even build centralized storage utilities that remote servers could access using LAN-free ISLs running over dark fiber connections.

"Our initial exploitation of SANs was purely connectivity and volume purchase-driven," says Durnford, adding: "Now we've got to consolidate onto fewer SANs that we can view and manage more simply."

RBC is certainly not alone in attempting to stem SAN island proliferation by consolidating onto fewer, larger fabrics. According to estimates by Gartner Inc., SANs with a port count of 320 or greater will grow at a rate of 40% over the five-year period ending in 2007. That compares to a growth rate of 12% for SANs of all sizes over the same period.

Simplify storage provisioning
Like RBC, many organizations consolidating storage onto larger SANs are hoping they can simplify storage provisioning.

"Next to backup, provisioning is the number one issue that IT is dealing with. At most companies, it's in the dark ages," says Arun Taneja, founder and consulting analyst of the Taneja Group in Hopkinton, MA. "Usually, you've got the systems administrator, database administrator and the storage administrator trying to figure out why an application isn't running fast enough. By the time they figure out they need more storage and go through the process of buying and deploying it, what should have taken a few minutes, takes weeks. The move from direct-attached storage to SANs was supposed to fix that, and it has made things easier," says Taneja, adding, "but not easy enough."

In Phase 1 of RBC's SAN consolidation push, a first step was to establish standard SAN deployment, management and procurement processes and encourage the company's lines of business to use them.

Eighteen months ago, Durnford's group created a storage steering committee, which included IT and business representatives from each of the company's businesses.

The purpose of the committee, says Durnford, was to announce SAN management and consolidation best practices, such as when you're consolidating SANs; how and in what sequence you hook up the switches; how you document the changes and how you back up data during the change and restore it. "We wanted to make sure we shared our experiences so that they could financially benefit from them, avoiding costly mistakes," Durnford says.

And authority over storage still remained with the lines of business. But with business executives and IT managers on the steering committee, Durnford was then able to generate business buy-in for storage consolidation.

Interoperability testing
Among the key best practices that Durnford's group emphasized was the need to thoroughly test and verify the interoperability of SAN elements, including switches, directors, arrays, etc.--before putting consolidated SANs into production. When in 1999, the technical support division started evaluating SANs, Durnford's group tested interoperability of various devices at different release levels from different vendors. They soon learned to anticipate compatibility problems, says Durnford, even in cases where vendors had certified interoperability between their devices. In a consolidated SAN, interoperability testing is particularly important.

"It doesn't matter what the vendor tells you, oftentimes when you hook up two switches that are not of agreeable [release] levels, you'd get what providers like to call 'unexpected results,'" says Durnford. "In fact, oftentimes when we heard from the vendor that it should work, it was a clue to us that we needed to test it, certify it and lock down the firmware."

Centralizing information about what devices and firmware releases actually work with each other and disseminating it to the rest of the organization, says Durnford, accelerates SAN consolidation. The technical support division also developed standard change management procedures that should be used when deploying and consolidating SANs.

Just-in-time purchasing
As part of Phase 1 of its plan, RBC standardized its SAN hardware vendors and then centralized storage procurement with Durnford's group. On the Windows side, for example, RBC will use Dell Computer Corp., EMC and IBM gear. On the Unix side, it will be EMC and IBM. Boiling down vendors to just this group will greatly reduce compatibility problems as more SAN consolidation projects take place. And centralizing procurement allows RBC not only to make sure all groups are buying only from the approved list, but it will also allow for much speedier provisioning.

RBC has begun what Durnford calls "just-in-time" provisioning. What this means is that vendors ship switches and disk arrays to RBC just before they are really needed. Durnford's group then configures them and tests them in advance. Only once a SAN has reached an agree-upon threshold of consumed storage space is the additional disk deployed. And only then does the company actually pay for it. Using this approach, Durnford says, RBC has been able to cut the time it takes to provision new SAN storage from up to eight weeks to, in some cases, in only 24 hours. (For a look at a different SAN consolidation project also intended to simplify provisioning, see "AXA Group's major consolidation push")

Using these procedures and standards, RBC businesses have begun to consolidate their existing SANs. In one case, six older Windows-based SANs in three different sites--totaling 11TB--are being consolidated into three SANs. In another, multiple SANs supporting 50 Exchange servers deployed across two sites are being consolidated along with the application servers.

RBC is also consolidating the mainframe side, mainly just to reduce software license charges. Last year, the company consolidated the North Carolina data center supporting its United States-based Centura banking business to an existing Toronto data center. Another consolidation project is under way, this time RBC's West Coast data center is merging with its Toronto site.

For now, says Durnford, RBC plans to continue consolidating SANs within each operating environment without creating central SANs that can support servers running different operating systems. Currently, says Durnford, the company is shooting for creating standard SAN fabrics of 500 ports. Although there are few technical reasons to avoid creating consolidated heterogeneous SANs, RBC is wise to focus first on consolidating within operating environments, says Taneja.

"There is LUN masking software for switches and HBAs that allow you to consolidate storage for different operating systems onto one SAN fabric, but organizations should probably think twice before attempting that," says Taneja.