Show-me state shows how to consolidate storage

Whether it's the result of a merger or just good housekeeping, at some point in time storage managers will have a storage consolidation project. The Missouri state government embarked on a major storage consolidation project that included numerous political and technical hurdles.

This Content Component encountered an error
This article can also be found in the Premium Editorial Download: Storage magazine: Salaries rise, but storage jobs get tougher:

The Missouri state government embarked on a major storage consolidation project that included numerous technical and political hurdles.

When matt blunt ran for the Missouri governorship in 2004, he made IT consolidation a major piece of his platform for saving money. Dan Ross, a career Missouri bureaucrat, saw candidate Blunt make that promise on the news and turned to his wife, remarking: "What a monster of a task for some poor fool!"

A few weeks after Blunt became governor, he called Ross and offered him the consolidated CIO job. "At first I thought he had misspoken and meant to offer me another job," recalls Ross, who had no serious IT experience. "I wasn't even what you would call a 'power user,'" he adds. It didn't matter, the governor told him; the CIO job needed someone with Ross' top administrative skills. He took the job.

The monster job consisted of consolidating the IT operations of 16 state cabinet agencies--each with independent IT budgets, staffs and decision making--into the state's central mainframe data center, which was wrapping up a mainframe consolidation effort. At the time, Ross calculated his consolidated IT budget would be $205 million in 2005 once he painstakingly pulled "well-camouflaged money into a visible pile," he says.

Today, the consolidated IT budget stands at $218 million. It should be closer to $250 million, but legislators have repeatedly dipped into the budget to redirect anticipated consolidation-related savings to other projects. "That's just too much money not to attract legislator interest," says Ross, who spends much of his time riding herd on this aggregated budget drawn from 121 separate funding sources and 161 appropriations, many of which are earmarked for specific programs--in short, an accounting nightmare.

The actual technical consolidation fell to the central data center staff. Howard Carter, the data center director, had a mixed reaction when he heard then-candidate Blunt's promise of big savings from IT consolidation. "I thought it could be good for state government, but I knew it would be a big increase in our workload," he says.

Multiple data centers
Nobody truly knows how much storage the 16 agencies had at the outset. Even now, some agencies may keep storage tucked in servers or as small SANs hidden from sight. At the start of the project, the data center managed 12TB of open-systems storage, reports Carter. Today, three years later, it hosts more than 200TB of open-systems storage, 10 times more than for mainframe operations.

To achieve the promised savings, the data center team has to squeeze every possible dollar out of its storage acquisitions. Two rules determine every choice: an agency can have whatever it wants for storage if it can pay for it, and dollars drive every decision. To date, "the consolidation activities have or will result in a total savings and cost avoidance of $52.3 million," reports Ross.

The Missouri state data center itself is the result of a consolidation effort that began years before Blunt made IT consolidation a priority. The initial consolidation involved only the state's biggest mainframe users. A second mainframe consolidation in 1996 pulled in the last of the state's mainframes. The current mainframe is IBM Corp.'s biggest--the System z10 configured for almost 6,000 general-purpose MIPS and another 3,000 MIPS from various assist processors.


Colorado's consolidation plan
In 2008, the Colorado legislature passed a bill to consolidate the IT functions of 13 executive agencies to produce an IT group of 1,100 people with a $250 million budget. The consolidation, intended to roll out over four years, will be a logical consolidation leaving much of the IT physically dispersed. Led by state CIO Mike Locatis and deputy CIO John Conley, the team began extensive planning and preparation even before the bill became effective. Initial planning steps included the following:
  • Peer organizations. Spoke with counterparts in Missouri and other states.


  • IT employee input. Received input from 950 of the 1,100 employees to date, often at town meeting-styled gatherings and through a Wiki.


  • Attorney general. Clarified employment issues.


  • Agency heads. Discussed service-level agreements, as well as IT budgeting and billing.


  • Vendors. Sent new procurement procedures and expectations to product vendors in an effort to build committed relationships.


  • Asset inventory. Identified technology on hand, end-of-life state, lease expiration and refresh plans.
"If I were doing this again, I would take even more time to listen to employees," says Conley. "This is a massive change for employees."

Open-systems storage growth
By the time the open-systems consolidation began, following Governor Blunt's election in 2004, "we had just a smattering of open systems in the data center, maybe a couple of dozen," says Carter. That amounted to approximately 12TB of storage.

The initial couple of dozen servers have grown to more than 500 consolidated servers, approximately 150 of which are virtual. When the consolidation began, the CIO's office had identified 1,400 servers. Many consolidated servers run as IBM server blades in the central data center, while others run as rack-mounted Dell Inc. servers. The standard server is a quad-core, dual-processor, dual-sockets machine, although a few four-processor, quad-core servers have been added.

For the open systems, the data center brought in IBM's SAN Volume Controller (SVC) and set up a SAN behind SVC. Storage currently consists of two IBM DS8000 arrays and five IBM DS4700 arrays, reports David Kassabaum, the data center's internal consultant (which is a state-employee position).

The DS8000 arrays are used for mainframe and open-systems storage with one DS8000 used exclusively for open systems. One DS8000, with 25TB of Fibre Channel (FC) storage consisting of 146GB drives, handles mainly mainframe storage with only 1.5TB of open-systems storage. The other DS8000, with 50TB of FC disk, handles only open-systems data. The rest of the storage is divided among the five DS4700 arrays, three designated for SATA only and two for FC. The five DS4700 arrays are used exclusively for open-systems storage.

With the consolidation, data center storage has been growing at a compound annual growth rate of 25%. The growth comes mainly from the open systems, which run Windows. "We haven't added new mainframe applications, although the existing applications continue to grow," says Carter.

To accommodate the open-systems storage traffic, the data center boosted the network fabric. It brought in two Cisco Systems Inc. 9513 switches, increasing the port count to 300 ports, notes Kassabaum.

Implementing the consolidation
Over the last three years, budgets have been consolidated and staff transferred to the CIO's office. Each agency has also determined what it needs for servers and storage. Some servers are now completely administered through the consolidated data center, while others reside in computer rooms across the state and are managed locally.

Old equipment is classified as surplus and sold whenever possible. Each agency's workload was put on new servers and storage. Surprisingly, very little storage and data was actually transferred to the data center. A big piece was moving and consolidating the Exchange email application and data; that migration was handled by an outside service company. In other cases, the agency's IT staff simply copied over the data.

At the data center, storage resides behind SVC, which allocates the storage as Vdisks, which are equivalent to LUNs. "The storage is allocated based on the agency's request and budget," says Kassabaum. "The agency pays for its storage."

Agencies have a choice of 15,000 rpm FC disk drives or 7,200 rpm SATA drives. The state's standard FC disk is 300GB and SATA drives are 750GB. Both drive types are configured for RAID 5.

The five DS4700 arrays are either FC or SATA. "We tried to mix FC and SATA, but it didn't work," says Kassabaum. The problem revolved around incompatibilities in the microcode. It was faster and easier just to segregate the different drive types.

Although the data center had been a longtime IBM mainframe shop, IBM still had to compete for the consolidated storage. "We looked at the other vendors where we had a state contract. It came down to the DS4700 or the EMC Clariion," says Kassabaum. SVC actually limited the choices, so whatever they chose had to work with SVC.

The capacity of each array was determined by pricing trends. "Everything is cost driven. We aim to get the best price per terabyte," says Kassabaum. As a result, the storage team often finds itself buying more storage than its immediate need. "But we always end up using it," he adds.

The data center ran an informal negotiation, not a formal bid process, which gave EMC multiple chances to compete against IBM and come up with a lower bid. Each time EMC was more expensive. Both machines did what the state wanted, other features were superfluous bells and whistles that didn't impact the selection. "Dollars drove every decision," says Kassabaum.

SVC proved instrumental. The staff relies on SVC to stripe the data across multiple arrays, not just multiple disks. SVC also includes a large upfront cache. Between the multi-array striping and the large cache, SVC allows the staff to coax better performance out of the SATA disks, reports Kassabaum.

The staff also relies on IBM's TotalStorage Productivity Center (TSPC) to do whatever storage management they perform. With TSPC, they can see how the storage is allocated among agencies and the size of the Vdisks. "If it gets really busy, we might move storage around to reduce bottlenecks," says Kassabaum. Otherwise, they don't optimize for storage performance.

Allocating storage
"The only tiering we do is tiering by dollars," says Kassabaum. If an agency doesn't pay for FC, they get SATA. Occasionally, Kassabaum and Patty Washburn, computer information technology specialist, will make a recommendation, but the IT staff assigned to the agency usually knows what it needs and, more importantly, what it's willing to pay.

Kassabaum and Washburn don't actively manage the storage beyond the basic allocation. They set up the client agency requests and leave the actual management of the storage to the assigned IT staff. "After we allocate the storage, we don't know how it's used," says Kassabaum. An agency might have 10TB allocated, but whether they're storing 1MB of data or 9.9TB, Kassabaum and Washburn won't know. As a result, they allocate storage at a very high utilization rate, typically 90% to 95%, but have no idea how much is actually being used.

"We're always adding storage. We've gone from zero to 200TB of open-systems storage in just a few years. Just yesterday I had a request for 2TB," says Washburn, adding that "our biggest challenge is keeping ahead of agency requests." Many requests are unexpected, leaving the team scrambling to come up with capacity.

The data center staff also relies on SVC to move data around. "When a customer has really sensitive data, we use SVC to move it to an enterprise [DS8000] box," says Kassabaum. To ensure security, the storage teams map each LUN or Vdisk to a single host and zone the fabric so a designated host can see only its LUN.

The data center relies on IBM's Tivoli Storage Manager (TSM) for backup and recovery. Backups are made daily to tape and shipped offsite. Ross recently signed an outsourcing contract with IBM for a remote hot site for the recovery of the mainframe and those open-system servers running mission-critical apps.

The biggest technical challenge turned out to be not the servers or storage but the network. "We didn't have the luxury of planning the fabric," says Kassabaum. The state wanted to capture the savings from consolidation as fast as possible, so the data center team began with the older switches they had, 34-port McData (now owned by Brocade) switches.

They didn't get very far along when the data center began running out of ports. "We couldn't just add more switches because we would encounter hop counts that were too large," explains Kassabaum. That's when they turned to the big Cisco switches.

Playing politics
Technical challenges, however, pale in comparison to the financial and political challenges. The politics of IT consolidation in any organization can be fierce, with jobs and budgets at stake. The Missouri IT consolidation began as a political campaign promise. When Ross saw it as a monster, he was thinking about the politics involved, not the technical changes.

"It would be very hard for a CIO from the outside to come in and do this. My strength lay in being a long-term bureaucrat who had worked for both parties," says Ross. "This [consolidation] doesn't have a lot to do with technology," he adds, it has more to do with budgets, funding, appropriations and accounting.

Ross immediately brought in three top deputies to help him. One was a strategic planner, the second was the financial wizard and the third, Chris Wilkerson, was the true technologist.

Resistance to consolidation was expected from the start. "Yes, you lose autonomy, but there's no loss of [IT] service," says Ross. Resistance was about control, not service delivery. Ross made sure that every agency's service-level agreement was honored or they negotiated changes. Unlike past CIOs, who could make recommendations for change but had no power, Ross came to the task with the full power of the Governor's office behind him. He could transfer budgets and staff as required.

Jobs were another concern, as civil service employees are protected. Ross opened all top IT positions for anyone to apply. Some staffers chose to retire. "Nobody would lose their job due to the consolidation. No one was laid off," says Ross.

However, people would be asked to change and many would have to learn new IT skills. "We had one person who wouldn't take new training," recalls Ross. Without training, they couldn't do the job and were gone. "If they wanted job security they had to be prepared to learn new skills and change," he says.

The IT consolidation drove the demand for and growth of IT. But the consolidated IT organization is now 51 positions smaller than when Ross started. "We were able to leverage technology to keep staffing down," he says, which is good because Governor Blunt capped overall Missouri government at 60,000 employees. The consolidated IT organization has 1,186 employees.

Agencies also played politics with the savings they would see from IT consolidation. "We automated a process that saved one agency $500,000 in postage expenses," says Ross. But the agency had to spend those savings before legislators grabbed it for use elsewhere (see "Lessons learned," below).


Lessons learned

  • Surround yourself with the right people. Designate outstanding people to handle critical planning, strategy, financial and technology tasks.


  • Consolidate as fast as possible. Changes in leadership mean there's a chance consolidation may never be completed or that it might get rolled back.


  • Preserve budget flexibility. Avoid getting locked into single-purpose budget allocations.


  • Expect rapid fabric growth. Avoid running out of ports or excessive hop counts.


  • Guard your consolidation savings. People will grab any available money they can identify.


  • Pay attention to politics. Technology is only one part, often the smaller part, of the consolidation challenge.


  • Consider skill sets. Jobs and responsibilities will change, often requiring training.

Future direction
Ross and Carter are looking ahead to the looming retirement of a number of veteran data center staff. Ross has gone so far as to set up an IT recruitment storefront on Second Life, the 3D virtual world. "I'm 58 years old and I never thought I'd have an avatar, but I do," says Ross, whose avatar, Second Life's slim, muscular male default avatar, along with avatars for other top IT executives, is busy recruiting technology people. They've already hired one techie and as the retirements start hitting, they'll be pounding the virtual pavements of Second Life and elsewhere for more hires.

By the end of summer 2008, a few of the agencies in the Missouri IT consolidation project had yet to be combined. With an election looming this month and Governor Blunt choosing not to run, the consolidation project will lose its biggest and most powerful champion. There's a possibility the new administration won't continue the consolidation or that some agencies may take it as an opportunity to bolt.

"But it will be hard to go back," says Carter. Ross, who has been reciting the mantra of budget savings and cost avoidance, expresses confidence that it won't be rolled back. Still, it's all politics, so anything can happen.

This was first published in November 2008

Dig deeper on Enterprise storage, planning and management

Pro+

Features

Enjoy the benefits of Pro+ membership, learn more and join.

0 comments

Oldest 

Forgot Password?

No problem! Submit your e-mail address below. We'll send you an email containing your password.

Your password has been sent to:

-ADS BY GOOGLE

SearchSolidStateStorage

SearchVirtualStorage

SearchCloudStorage

SearchDisasterRecovery

SearchDataBackup

Close