A million-dollar disaster provided a reality check for Mesaba Aviation Inc. Before the disaster, the Minneapolis,...
Minn.-based airline had balked at updating its storage infrastructure. Then, in December 2000, a major power outage destroyed Mesaba's data -- data that was not backed up.
The outage downed the single server that handled the flight-tracking system for the Northwest Airlines affiliate, according to Scott Ficek, Mesaba's director of IT. When the server was brought back online, it was discovered that the data was corrupted. With no backup data, all aircraft and crews were grounded for more than four hours. The outage and subsequent scramble to reconstruct the files needed to run the airline caused the cancellation of 329 flights and the delay of 399 flights. As a result, the company lost close to one million dollars in revenue.
Ironically, Ficek's department was in the research phase of an infrastructure redesign project when the disaster struck.
"We had already started looking at storage area networks [SANs] and completed initial evaluations and investigative work," Ficek said.
The massive disruption of the system pushed the acquisition of a new system from priority to necessity. Mesaba was running two independent, mutually exclusive servers at the time of the outage. One was a Compaq server running the airline's flight-tracking systems. The second was an "extremely old" Hewlett-Packard server, which ran the crew-tracking systems.
"A new system had been a tougher sell when it was just simply that 'we need more storage.' That was hard to quantify," Ficek said. "Before experiencing a million dollar loss of revenue, it was harder to calculate an ROI on consolidating our storage."
With a real-life ROI calculation in hand, Ficek cleared the way to update storage. High availability and disaster recovery topped Ficek's purchasing criteria for a new storage system.
It was clear to Ficek that the airline needed centralized storage. Evaluations completed before the disaster had convinced him that SAN was a better option than network-attached storage (NAS). Real-life examples also tipped the scale toward a SAN: Ficek had seen other airlines experience poor results after using flight-tracking systems similar to Mesaba's in a NAS environment.
Ficek and his IT team tested SAN solutions from EMC Corp. of Hopkinton, Mass., and Eden Prairie, Minn.-based XIOtech Corp., a wholly-owned subsidiary of Seagate Technology Inc. of Scotts Valley, Calif.
After prolonged discussions with both vendors, Mesaba's IT team favored XIOtech.
"We were impressed that the XIOtech group understood the Wintel market," Ficek said. "The XIOtech SAN was not a downsized version of a mainframe system trying to fit into a Unix or Wintel arena."
XIOtech also came out on top because its staff includes a respectable number of certified Novell engineers. Mesaba's primary network runs on a Novell operating system, and Ficek planned to achieve high availability with Novell cluster products.
XIOtech fit the bill in terms of disaster recovery capabilities with remote mirroring technology. Ficek noted EMC didn't have a remote mirroring option when Mesaba was evaluating products.
For the first phase of the upgrade, Mesaba's IT team chose XIOtech's Magnitude hardware platform. Ficek endorsed the choice after having found that Magnitude incorporates all the components of a SAN in one centralized, manageable and highly available configuration. The clincher for Ficek was Magnitude's compatibility with clustered servers.
Initially, Mesaba built a cluster using three Compaq servers running Novell 5.l and Cluster 1.0. The fact that Mesaba's applications run on a database made by Austin, Texas-based Pervasive Software Inc. posed a problem. Pervasive had never run its database in a clustered Novell environment.
"We were actually getting fresh code from the Pervasive developers as we were testing to fix problems," Ficek said.
Another bug in the implementation was the technical limitation of Gadzoox Network Inc.'s Fibre Channel switches, Ficek's first choice for connecting the servers. He recently replaced those switches with Brocade switches, which are running smoothly.
To achieve strong disaster recovery capabilities, Mesaba needed off-site backup. The company decided to use part of a training facility about 10 miles away from headquarters. Ficek's team built a backup facility within this building wherein data would be safe if a disaster occurred at the main site.
A second Magnitude was purchased for the remote site. Each Magnitude was connected to an IP network via a Nishan Systems 3300 Multiprotocol Storage Switch and an OC-3 SONET data connection. XIOtech's REDI SAN Links Replicator software is used to ensure accurate replication of mission critical data.
The result is a complex system that is simple to use and easy to manage, Ficek said. Now, he can push the power button on a production server, and watch the application shift from one server to another. This process is completed without users experiencing anything more than a five- to 10-second pause in their applications. To Ficek, "that's really amazing!"
Click for more information on Mesaba Aviation Inc.
Learn more about XIOtech here.