As electronic medical records proliferate within the U.S. healthcare system, one hospital has overhauled its tiered...
storage infrastructure to accommodate growing volumes of highly sensitive and mission-critical data using a combination of products from IBM and Symantec Corp.
Mississippi Baptist Health Systems senior systems engineer Jim Touchstone said the project to support explosive data growth at the hospital began with backup in 2005. At the time, the hospital was using Symantec's Backup Exec, but "we'd become way too big for that product," Touchstone said. "We went with NetBackup 6.0 on Symantec's recommendation." Symantec and IBM also worked together to recommend the right IBM server to support the new backup implementation.
From there, Touchstone said the collaboration between Mississippi Baptist Health Systems, IBM and Symantec continued, soon spreading to cover the entire four-tier data storage environment. The hospital worked with its vendors to eventually put together its own integrated data center stack designed to reduce the manual process of continually provisioning storage for new hosts and send less active data to lower tiers of storage. This was completed years before the recent trend of storage vendor consortiums pre-integrating such infrastructure stacks.
Touchstone said the current hospital system supports more than 500 servers -- most of them virtualized using VMware -- with a 77:1 virtual-to-physical server ratio. Data first flows from hosts through Symantec's Storage Foundation, which allows volumes to exceed the typical size compatible with operating systems without disrupting performance. From there, it goes to an IBM SAN Volume Controller (SVC), which is attached to an IBM DS8300 array containing tier 0 and tier 1 disk, which holds the most mission-critical files while they remain active.
The SVC is also attached to three more IBM disk arrays, an N5200-G20 multiprotocol storage system rebranded from NetApp Inc., as well as an IBM DS4400 and DS4200. The N-series array is used for nearline retention of PACS system data, while Symantec's Enterprise Vault software and NetBackup Disk Staging Storage Unit (DSSU) feature store disk-based backup data on the DS4000 series arrays. When data hits the DS4000 arrays, the DSSU sends new data to an IBM TS3584 tape library, which Mississippi Baptist Health Systems calls tier 3. The SVC allows data to move non-disruptively between the disk tiers.
The Storage Foundation and SVC storage virtualization products are transparent to each other, and allow the hospital to swap new operating systems, server hardware, and applications in and out of the infrastructure without having to re-optimize the storage infrastructure.
"If we go to a certain version of a new OS, the native drivers and the SVC are on the same sheet of music," Touchstone said. The flexibility and built-in alerting and reporting have meant the hospital hasn't had to hire additional engineers to manage some 200 TB of data that is growing every day.
The tier 3 library is located across the street from the health center, and uses self-encrypting TS1130 tape drives. Touchstone said this is considered the "fast access library," which allows files to be recalled quickly when doctors need them. For example, one doctor called a heart catheterization file back from the "farline" library in a minute and eight seconds recently. "That's hauling the mail for tape," Touchstone said.
Ultimately, files that must be retained for seven years or more for compliance with state and federal regulations are stored, encrypted, on another tape library across town from the health center.
New challenges for healthcare IT on the horizon
Touchstone praised the integration work done by his vendors, but no sooner have these first phases of updating the infrastructure been completed than a new phase will begin — setting up a disaster recovery (DR) environment. The health system is currently looking at the costs of building out a second data center.
And while the hospital has its infrastructure in place for now, "the sheer volume of data accumulating due to federal and state regulations is our biggest challenge," Touchstone said. It's not just simple data growth, but the criticality of medical records and remaining compliant that make accommodating increased use of electronic medical records an uphill battle. "It all has to be done at the speed of 'right now' because someone's life depends on that," he said. "This is a life and death industry, and you have to get data back in a timely manner, and it has to be correct."
To keep up with performance requirements, Touchstone said he's hoping to see solid-state drives (SSDs) become more affordable, and he's eager to add the IBM Easy Tier software when it becomes available for the DS8300 and SVC. IBM supports SSDs in the DS8000 series, but Touchstone said he'll wait for Easy Tier for more efficient management. "Otherwise, [with SSDs and HDDs intermingled], the array might make SSDs hot spares" instead of using the Flash capacity for hot data. IBM's support documents recommend customers not mix SSD and hard disk drive capacity under the same Device Adapter pair.