It sounds like the plot for a bad movie. A college student works tirelessly to finish a big exam or a thesis and at the last second the computer fails and everything is lost. Well, this nightmare became a reality at Cuesta College when a storage array failed and 100% of the student data stored on the network was lost.
"We had an array on our SAN that had a failure and when it failed it took all our data with it," said Jay Chalfant, supervisor of PC and network support at Cuesta College, San Luis Obispo, Calif. "It took us a very long time to recover from the outage. We took a major hit."
When the SAN went down, Cuesta was unable to access any of its primary SQL server data. It lost all of its domain controllers, except for one for faculty members and one for students, and couldn't recover any data for its computer labs or classroom PCs.
Chalfant said the array failure had a serious impact on the more than 10,000 students and faculty members at the college and monopolized his IT staff for two weeks. "We had two techs working close to 24 hours a day. Everything else came off their schedule for two weeks. We were feeling the residual effects up to two months later," he said.
Chalfant would not discuss which vendor supplied the array, but SearchStorage.com has learned that Cuesta College was running a Virtual Array from Hewlett-Packard Co. This system has since been discontinued.
Grant Chesy, an IT administrator with Cuesta, said they made every effort to right the ship. "It's unfortunate that we couldn't have confidence in this particular array. It was an expensive piece of storage that had reliability issues. I don't think the vendor's arrays are bad on the whole. They ate a big bill when they didn't have to and helped us a lot," he added.
To protect against another total data loss, the Cuesta team researched mirroring solutions to move data to another storage device and have a buffer zone in case the SAN failed again.
Chalfant said he looked at software companies like Veritas Software Corp., but did not want the hassle of installing software agents on every host. Ultimately, he settled on an appliance from FalconStor Software Inc., Melville, New York, to get the job done.
But unfortunately Cuesta's problems didn't end there. Soon after FalconStor's IPStor product was online, the SAN failed again. But rather than a few weeks of interrupted data services, the network was down for a few hours.
Cuesta and FalconStor were tightlipped on how much the IPStor implementation cost the college, but IPStor is priced as a base server, plus optional service modules. Agents and some experts believe the implementation carries a relatively high price tag.
"Sometimes we have limitations on hiring staff because we are publicly funded. What we spent was clearly justified just to avoid a hit to our staff and resources that those kinds of [array] failures can cause," Chalfant said.
The FalconStor implementation took a little longer than normal due to a few minor interoperability bumps. Chalfant said FalconStor had to work through some issues in getting the HP array to work with IPStor. "We're the only FalconStor customer using that line of virtual arrays. They had developers working it out and, when they came to install [IPStor], they had come up with a patch," he said.
Once IPStor was installed, Cuesta mirrored all of its 2 terabytes of data on the SAN between multiple HP storage arrays managed by IPStor. A subsequent partial failure of that same array yielded no interruption to the college's data services. FalconStor's storage architects worked with Cuesta to get a handle on its weekend backup windows, which were bleeding into Monday.
IPStor sits in the middle of the data path on a server and supports SCSI, iSCSI and Fibre Channel-attached disk and tape devices. IPStor is now managing the storage of 20 servers in Cuesta College's SAN, including HP Netserver and Compaq ProLiant systems running Active Directory, DNS, Clustering Services and SQL 2000 over the Windows 2000, Windows Server and HP-UX platforms.
For more information: