This article can also be found in the Premium Editorial Download "Storage magazine: Salaries rise, but storage jobs get tougher."
Download it now to read this article plus other related content.
Implementing the consolidation
Old equipment is classified as surplus and sold whenever possible. Each agency's workload was put on new servers and storage. Surprisingly, very little storage and data was actually transferred to the data center. A big piece was moving and consolidating the Exchange email application and data; that migration was handled by an outside service company. In other cases, the agency's IT staff simply copied over the data.
At the data center, storage resides behind SVC, which allocates the storage as Vdisks, which are equivalent to LUNs. "The storage is allocated based on the agency's request and budget," says Kassabaum. "The agency pays for its storage."
Agencies have a choice of 15,000 rpm FC disk drives or 7,200 rpm SATA drives. The state's standard FC disk is 300GB and SATA drives are 750GB. Both drive types are configured for RAID 5.
The five DS4700 arrays are either FC or SATA. "We tried to mix FC and SATA, but it didn't work," says Kassabaum.
| The problem revolved around incompatibilities in the microcode. It was faster and easier just to segregate the different drive types.
Although the data center had been a longtime IBM mainframe shop, IBM still had to compete for the consolidated storage. "We looked at the other vendors where we had a state contract. It came down to the DS4700 or the EMC Clariion," says Kassabaum. SVC actually limited the choices, so whatever they chose had to work with SVC.
The capacity of each array was determined by pricing trends. "Everything is cost driven. We aim to get the best price per terabyte," says Kassabaum. As a result, the storage team often finds itself buying more storage than its immediate need. "But we always end up using it," he adds.
The data center ran an informal negotiation, not a formal bid process, which gave EMC multiple chances to compete against IBM and come up with a lower bid. Each time EMC was more expensive. Both machines did what the state wanted, other features were superfluous bells and whistles that didn't impact the selection. "Dollars drove every decision," says Kassabaum.
SVC proved instrumental. The staff relies on SVC to stripe the data across multiple arrays, not just multiple disks. SVC also includes a large upfront cache. Between the multi-array striping and the large cache, SVC allows the staff to coax better performance out of the SATA disks, reports Kassabaum.
The staff also relies on IBM's TotalStorage Productivity Center (TSPC) to do whatever storage management they perform. With TSPC, they can see how the storage is allocated among agencies and the size of the Vdisks. "If it gets really busy, we might move storage around to reduce bottlenecks," says Kassabaum. Otherwise, they don't optimize for storage performance.
Kassabaum and Washburn don't actively manage the storage beyond the basic allocation. They set up the client agency requests and leave the actual management of the storage to the assigned IT staff. "After we allocate the storage, we don't know how it's used," says Kassabaum. An agency might have 10TB allocated, but whether they're storing 1MB of data or 9.9TB, Kassabaum and Washburn won't know. As a result, they allocate storage at a very high utilization rate, typically 90% to 95%, but have no idea how much is actually being used.
"We're always adding storage. We've gone from zero to 200TB of open-systems storage in just a few years. Just yesterday I had a request for 2TB," says Washburn, adding that "our biggest challenge is keeping ahead of agency requests." Many requests are unexpected, leaving the team scrambling to come up with capacity.
The data center staff also relies on SVC to move data around. "When a customer has really sensitive data, we use SVC to move it to an enterprise [DS8000] box," says Kassabaum. To ensure security, the storage teams map each LUN or Vdisk to a single host and zone the fabric so a designated host can see only its LUN.
The data center relies on IBM's Tivoli Storage Manager (TSM) for backup and recovery. Backups are made daily to tape and shipped offsite. Ross recently signed an outsourcing contract with IBM for a remote hot site for the recovery of the mainframe and those open-system servers running mission-critical apps.
The biggest technical challenge turned out to be not the servers or storage but the network. "We didn't have the luxury of planning the fabric," says Kassabaum. The state wanted to capture the savings from consolidation as fast as possible, so the data center team began with the older switches they had, 34-port McData (now owned by Brocade) switches.
They didn't get very far along when the data center began running out of ports. "We couldn't just add more switches because we would encounter hop counts that were too large," explains Kassabaum. That's when they turned to the big Cisco switches.
This was first published in November 2008