Published: 10 Nov 2008
| The Missouri state government embarked on a major storage consolidation project that included numerous technical and political hurdles.
When matt blunt ran for the Missouri governorship in 2004, he made IT consolidation a major piece of his platform for saving money. Dan Ross, a career Missouri bureaucrat, saw candidate Blunt make that promise on the news and turned to his wife, remarking: "What a monster of a task for some poor fool!"
A few weeks after Blunt became governor, he called Ross and offered him the consolidated CIO job. "At first I thought he had misspoken and meant to offer me another job," recalls Ross, who had no serious IT experience. "I wasn't even what you would call a 'power user,'" he adds. It didn't matter, the governor told him; the CIO job needed someone with Ross' top administrative skills. He took the job.
The monster job consisted of consolidating the IT operations of 16 state cabinet agencies--each with independent IT budgets, staffs and decision making--into the state's central mainframe data center, which was wrapping up a mainframe consolidation effort. At the time, Ross calculated his consolidated IT budget would be $205 million in 2005 once he painstakingly pulled "well-camouflaged money into a visible pile," he says.
Today, the consolidated IT budget stands at $218 million. It should be closer to $250 million, but legislators have repeatedly dipped into the budget to redirect anticipated consolidation-related savings to other projects. "That's just too much money not to attract legislator interest," says Ross, who spends much of his time riding herd on this aggregated budget drawn from 121 separate funding sources and 161 appropriations, many of which are earmarked for specific programs--in short, an accounting nightmare.
The actual technical consolidation fell to the central data center staff. Howard Carter, the data center director, had a mixed reaction when he heard then-candidate Blunt's promise of big savings from IT consolidation. "I thought it could be good for state government, but I knew it would be a big increase in our workload," he says.
Multiple data centers
To achieve the promised savings, the data center team has to squeeze every possible dollar out of its storage acquisitions. Two rules determine every choice: an agency can have whatever it wants for storage if it can pay for it, and dollars drive every decision. To date, "the consolidation activities have or will result in a total savings and cost avoidance of $52.3 million," reports Ross.
The Missouri state data center itself is the result of a consolidation effort that began years before Blunt made IT consolidation a priority. The initial consolidation involved only the state's biggest mainframe users. A second mainframe consolidation in 1996 pulled in the last of the state's mainframes. The current mainframe is IBM Corp.'s biggest--the System z10 configured for almost 6,000 general-purpose MIPS and another 3,000 MIPS from various assist processors.
Open-systems storage growth
The initial couple of dozen servers have grown to more than 500 consolidated servers, approximately 150 of which are virtual. When the consolidation began, the CIO's office had identified 1,400 servers. Many consolidated servers run as IBM server blades in the central data center, while others run as rack-mounted Dell Inc. servers. The standard server is a quad-core, dual-processor, dual-sockets machine, although a few four-processor, quad-core servers have been added.
For the open systems, the data center brought in IBM's SAN Volume Controller (SVC) and set up a SAN behind SVC. Storage currently consists of two IBM DS8000 arrays and five IBM DS4700 arrays, reports David Kassabaum, the data center's internal consultant (which is a state-employee position).
The DS8000 arrays are used for mainframe and open-systems storage with one DS8000 used exclusively for open systems. One DS8000, with 25TB of Fibre Channel (FC) storage consisting of 146GB drives, handles mainly mainframe storage with only 1.5TB of open-systems storage. The other DS8000, with 50TB of FC disk, handles only open-systems data. The rest of the storage is divided among the five DS4700 arrays, three designated for SATA only and two for FC. The five DS4700 arrays are used exclusively for open-systems storage.
With the consolidation, data center storage has been growing at a compound annual growth rate of 25%. The growth comes mainly from the open systems, which run Windows. "We haven't added new mainframe applications, although the existing applications continue to grow," says Carter.
To accommodate the open-systems storage traffic, the data center boosted the network fabric. It brought in two Cisco Systems Inc. 9513 switches, increasing the port count to 300 ports, notes Kassabaum.
Implementing the consolidation
Old equipment is classified as surplus and sold whenever possible. Each agency's workload was put on new servers and storage. Surprisingly, very little storage and data was actually transferred to the data center. A big piece was moving and consolidating the Exchange email application and data; that migration was handled by an outside service company. In other cases, the agency's IT staff simply copied over the data.
At the data center, storage resides behind SVC, which allocates the storage as Vdisks, which are equivalent to LUNs. "The storage is allocated based on the agency's request and budget," says Kassabaum. "The agency pays for its storage."
Agencies have a choice of 15,000 rpm FC disk drives or 7,200 rpm SATA drives. The state's standard FC disk is 300GB and SATA drives are 750GB. Both drive types are configured for RAID 5.
The five DS4700 arrays are either FC or SATA. "We tried to mix FC and SATA, but it didn't work," says Kassabaum. The problem revolved around incompatibilities in the microcode. It was faster and easier just to segregate the different drive types.
Although the data center had been a longtime IBM mainframe shop, IBM still had to compete for the consolidated storage. "We looked at the other vendors where we had a state contract. It came down to the DS4700 or the EMC Clariion," says Kassabaum. SVC actually limited the choices, so whatever they chose had to work with SVC.
The capacity of each array was determined by pricing trends. "Everything is cost driven. We aim to get the best price per terabyte," says Kassabaum. As a result, the storage team often finds itself buying more storage than its immediate need. "But we always end up using it," he adds.
The data center ran an informal negotiation, not a formal bid process, which gave EMC multiple chances to compete against IBM and come up with a lower bid. Each time EMC was more expensive. Both machines did what the state wanted, other features were superfluous bells and whistles that didn't impact the selection. "Dollars drove every decision," says Kassabaum.
SVC proved instrumental. The staff relies on SVC to stripe the data across multiple arrays, not just multiple disks. SVC also includes a large upfront cache. Between the multi-array striping and the large cache, SVC allows the staff to coax better performance out of the SATA disks, reports Kassabaum.
The staff also relies on IBM's TotalStorage Productivity Center (TSPC) to do whatever storage management they perform. With TSPC, they can see how the storage is allocated among agencies and the size of the Vdisks. "If it gets really busy, we might move storage around to reduce bottlenecks," says Kassabaum. Otherwise, they don't optimize for storage performance.
Kassabaum and Washburn don't actively manage the storage beyond the basic allocation. They set up the client agency requests and leave the actual management of the storage to the assigned IT staff. "After we allocate the storage, we don't know how it's used," says Kassabaum. An agency might have 10TB allocated, but whether they're storing 1MB of data or 9.9TB, Kassabaum and Washburn won't know. As a result, they allocate storage at a very high utilization rate, typically 90% to 95%, but have no idea how much is actually being used.
"We're always adding storage. We've gone from zero to 200TB of open-systems storage in just a few years. Just yesterday I had a request for 2TB," says Washburn, adding that "our biggest challenge is keeping ahead of agency requests." Many requests are unexpected, leaving the team scrambling to come up with capacity.
The data center staff also relies on SVC to move data around. "When a customer has really sensitive data, we use SVC to move it to an enterprise [DS8000] box," says Kassabaum. To ensure security, the storage teams map each LUN or Vdisk to a single host and zone the fabric so a designated host can see only its LUN.
The data center relies on IBM's Tivoli Storage Manager (TSM) for backup and recovery. Backups are made daily to tape and shipped offsite. Ross recently signed an outsourcing contract with IBM for a remote hot site for the recovery of the mainframe and those open-system servers running mission-critical apps.
The biggest technical challenge turned out to be not the servers or storage but the network. "We didn't have the luxury of planning the fabric," says Kassabaum. The state wanted to capture the savings from consolidation as fast as possible, so the data center team began with the older switches they had, 34-port McData (now owned by Brocade) switches.
They didn't get very far along when the data center began running out of ports. "We couldn't just add more switches because we would encounter hop counts that were too large," explains Kassabaum. That's when they turned to the big Cisco switches.
"It would be very hard for a CIO from the outside to come in and do this. My strength lay in being a long-term bureaucrat who had worked for both parties," says Ross. "This [consolidation] doesn't have a lot to do with technology," he adds, it has more to do with budgets, funding, appropriations and accounting.
Ross immediately brought in three top deputies to help him. One was a strategic planner, the second was the financial wizard and the third, Chris Wilkerson, was the true technologist.
Resistance to consolidation was expected from the start. "Yes, you lose autonomy, but there's no loss of [IT] service," says Ross. Resistance was about control, not service delivery. Ross made sure that every agency's service-level agreement was honored or they negotiated changes. Unlike past CIOs, who could make recommendations for change but had no power, Ross came to the task with the full power of the Governor's office behind him. He could transfer budgets and staff as required.
Jobs were another concern, as civil service employees are protected. Ross opened all top IT positions for anyone to apply. Some staffers chose to retire. "Nobody would lose their job due to the consolidation. No one was laid off," says Ross.
However, people would be asked to change and many would have to learn new IT skills. "We had one person who wouldn't take new training," recalls Ross. Without training, they couldn't do the job and were gone. "If they wanted job security they had to be prepared to learn new skills and change," he says.
The IT consolidation drove the demand for and growth of IT. But the consolidated IT organization is now 51 positions smaller than when Ross started. "We were able to leverage technology to keep staffing down," he says, which is good because Governor Blunt capped overall Missouri government at 60,000 employees. The consolidated IT organization has 1,186 employees.
Agencies also played politics with the savings they would see from IT consolidation. "We automated a process that saved one agency $500,000 in postage expenses," says Ross. But the agency had to spend those savings before legislators grabbed it for use elsewhere (see "Lessons learned," below).
By the end of summer 2008, a few of the agencies in the Missouri IT consolidation project had yet to be combined. With an election looming this month and Governor Blunt choosing not to run, the consolidation project will lose its biggest and most powerful champion. There's a possibility the new administration won't continue the consolidation or that some agencies may take it as an opportunity to bolt.
"But it will be hard to go back," says Carter. Ross, who has been reciting the mantra of budget savings and cost avoidance, expresses confidence that it won't be rolled back. Still, it's all politics, so anything can happen.