To be sure, determining storage requirements and procedures for regulatory compliance is a group effort. "It really does require a close working relationship with the clinical units," says Daniel Morreale, CIO of the North Bronx Healthcare Network in New York City, describing his formula for success when dealing with HIPAA requirements. Morreale advises that IT take an active role and ask questions such as: "What [data] are you collecting; why are you collecting it; how will you need to see it now; and what do you anticipate your needs with this data are going to be down the road?"
Developing a written policy for all involved parties to sign off on is a critical step. In some cases, putting together a compliance plan might require interpreting dozens of regulations. For multinational companies, this task can be daunting. Lois Hughes, senior manager of business application systems at Tektronix, says her team had to understand the requirements of the dozens of countries where they do business to put together their retention system. "We have a central retention document that is maintained current for all 27 countries where we do business," says Hughes.
But even without the special demands imposed by a global business model, creating a policy can be taxing. Although his organization primarily heeds only to state regulations, David Taylor, CIO of the Florida Department of Health in Tallahassee, FL, says, "The most difficult thing in the project was developing policy, and getting all the people and partners to agree on the policy, rather than the technical implementation." As with most organizations faced with compliance issues, the Department of Health formed a working group: "We pulled together the legal staff, the HIPAA compliance staff, the security and privacy staff, as well as folks that were administering the system," says Taylor.
While preparing for compliance can be an arduous process, it can be regarded as an opportunity to finally get a handle on storage and data management. "Compliance shouldn't be seen as a corporate tax, but really as an opportunity--a strategic investment, actually," says ESG's Gerr, noting that an effective compliance effort also "helps organizations both improve their ability to manage and protect valuable information."
"First of all, it's good business," says John Halamka, CIO of Harvard Medical School and six affiliated hospitals, in describing his organization's HIPAA compliance efforts and the related benefits they discovered. "The timing was right--we could both achieve what we thought was essential for our users and meet what we imposed on ourselves as HIPAA reliability standards."
The vast majority of compliance rules, however, don't go that far. But as part of the process of ensuring that retained data can't be tampered with, many companies are opting for special storage systems such as EMC Corp.'s Centera, IBM Corp.'s TotalStorage Data Retention 450 and Network Appliance Inc.'s SnapLock. These storage systems effectively lock retained data, barring any modifications or deletions of records until predetermined retention periods have elapsed.
North Bronx Healthcare Network's Morreale says they took a broader approach when designing the storage system for the data that HIPAA requires they retain. Because the digital images they must retain are so voluminous, they added 280TB of various forms of EMC storage over the past two years. Morreale describes the lifecycle approach they took: "The more current stuff I'm keeping on my SAN [storage area network], and as we age it out, our intention is to move to NAS [network-attached storage], where it's not so transactional anymore, and then we're going to archive on our CAS [content-addressed storage]."
As the result of its compliance program, Harvard Medical spent about $2 million for additional storage, and also instituted a tiered-storage architecture with an eye to implementing information lifecycle management (ILM). Toward that end, Halamka put a system in place to prioritize 200 applications and their related data that started with the question, "If you take each application that we run, what are the demands for uptime, data integrity and recovery?" Based on this analysis, they were able to determine how to migrate data between their EMC Symmetrix and Clariion systems, and then to their StorageTek tape libraries. While they eschewed WORM tape, Halamka says that their medical images are stored on an EMC Centera device and then moved to tape.
State Street Global Advisors' Linden says the firm will expand its storage spending by approximately 30% to 40%. Linden sees compliance as an opportunity to implement ILM. "When I have to engage in a technology refresh, I look to use that as a funding mechanism to further our storage framework." He also figures that their increase in storage spending would have been greater without the ILM effort.
At Citigroup in New York City, meeting compliance requirements is an enormous undertaking because of the size of the company, the number of subsidiary companies and the corporation's varied financial businesses. For example, senior storage engineer Shaun
Mahoney says they've implemented an interim e-mail archiving solution using journaling and off-site storage. With 235,000 e-mail users, Mahoney says, "The scale of our e-mail environment prohibits us from using a lot of solutions at their present maturity levels." Citigroup is working with several e-mail archiving vendors to modify their programs so they can handle Citigroup's large number of Exchange users.
Of course, Citigroup's regulatory efforts go well beyond e-mail. "It's not just e-mail or just instant messaging--it's across the board," says Mahoney, adding, "I don't know of any business that has only one application that deals with financial markets." To cope with the storage requirements of compliance, Citigroup has already "a fairly sizable amount" of storage capacity to meet their interim requirements and the company expects to add significantly more, especially when a final e-mail solution is put in place.
Future access
Figuring out what to save and where to put it is just half of the compliance equation. The real acid test of a successful compliance program is being able to produce information when requested. This can be a bit dicey, even if the requested information is relatively new, but is in a form that makes it difficult to search for and find specific data. For example, a typical backup of an e-mail application might capture all the data, but it would be difficult to find individual messages related to a particular topic, for instance.
Software tools that archive e-mail and database applications not only take the sweat out of the task of archiving the data, but they also provide the means to quickly find discrete portions of information. The Florida Department of Health used to rely on backup tapes for saving Exchange data, according to CIO David Taylor. But while it protected the data, access was a problem. They installed KVS Enterprise Vault for Exchange and the benefits were almost instantly realized. Before KVS, Taylor says, they would have had to "pull all the backup tapes from our 20 Exchange servers that are distributed statewide and restore all those tapes" to rebuild an Exchange server and extract all the data, tape by tape. That effort could take over 1,000 man hours.
"With the KVS system," says Taylor, "we were able to restore 7,700 e-mails in under a 10-minute search, with another three hours or so of work to extract the messages. The e-mail that's archived by the KVS application is stored on an 8TB EMC Centera array rather than tape, which greatly facilitates access. Taylor expects the current Centera configuration will meet their needs for the next three years.
Tektronix uses OuterBay's Application Data Management (ADM) suite to manage the archiving of its multinational database applications. The LiveArchive module in the suite winnows databases of aging data based on user-defined policies and moves it to less expensive storage. Ultimately, OuterBay's application can migrate the records to an "encapsulated archive" where they are saved with the appropriate metadata using XML to preserve them as complete transactions.
Because OuterBay uses XML to archive the records, the information will be accessible even if the originating application is unavailable. Over the years, applications are likely to change or be replaced, and application data structures are likely to undergo modifications, too. In those cases, if data is kept in its native format, satisfying an information request might entail rebuilding old application environments. OuterBay's XML-based archival allows the old data to be accessed and read using any XML-capable application--even a Web browser. Princeton Softech's popular Active Archive suite of database archiving applications also offers an XML output option to help ensure future accessibility.
KVS takes a similar approach with Enterprise Vault by providing an option to store archived e-mail in HTML format. "We do that because keeping an HTML rendition gives you your best chance of future proofing," says Mary Kay Roberto, senior vice president at KVS. Roberto notes that over a period of a few years, systems are likely to be upgraded, which could make recovering e-mail that was created with earlier application versions difficult. Organizations that have to deal with long retention periods should give serious consideration to saving data in non-proprietary formats.
North Carolina State University, Raleigh, NC, uses Documentum to archive data in PDF format from its PeopleSoft applications to ensure that the information will be accessible during its seven-year retention period. Although it's a proprietary format, PDF is so widely used that it approaches de facto status as a standard for document exchange. While Documentum also supports XML and HTML formats, Henry Vail, systems architect at the university, says they opted for PDF to conform with state guidelines for storing nonalterable content. To further ensure the integrity of retained data, they store it on a NetApp SnapLock system, which replaced their overburdened optical storage setup.
Documentum--paired with an EMC Centera
array--is also the key to State Street Global Advisor's compliance program. "Those technologies coupled together have met the litmus test for various proofs in terms of validation of records." Linden cites the variety of output options that Documentum offers, and sees it as "a very solid fit with the rest of storage infrastructure."
Once an archiving system is in place--for e-mail, databases or file system content--it should be part of the compliance policy to periodically test it to ensure that data can be retrieved quickly. Testing should also be part of the process whenever host applications are upgraded.
Desktops and beyond
As companies scurry to meet compliance requirements, the immediate pain points are usually e-mail and corporate accounting systems, as these systems typically house the data that is required for retention. However, a growing amount of corporate data subject to retention regulations now resides outside the data center on desktop computers and at remote storage sites.
ESG's Gerr points out that applications such as Oracle Financials and PeopleSoft have internal auditing capabilities to keep track of documents produced by those applications. But data that's produced or altered outside the applications represents "an area that is woefully underserved." Gerr adds, "It's very, very difficult to protect the edge."
However, there are ways to address the desktop issue, including backing up all network-connected desktop systems using a product such as Connected's DataProtector/PC. DataProtector can be launched locally or set to periodically back up desktop machines; it only backs up changed data and doesn't store data duplicated on multiple machines, so its effect on performance should be minimal.
Some companies protect desktop data by enacting policies that use logon scripts to ensure that all documents are saved to network drives rather than locally. It's company policy at Citigroup, according to Shaun Mahoney, to store desktop data centrally and they're now addressing the issue of remote PCs. "We're looking at ways of working remotely that centrally store the information whether it's thin clients or VPN connections that provide a way to access that data centrally."
The Florida Department of Health takes a similar approach. Says CIO Taylor: "We don't permit, to the best of our ability, people storing any data locally--it's all on their own network shares." For laptops, they rely on users copying their data to central storage when they reconnect to the network. As part of their HIPAA compliance effort, North Bronx Healthcare Network has also banned desktop storage, diverting all data to networked storage in a physically secured data center.
Other considerations
In the future, the question of what to save will undoubtedly become more problematic, especially when considering new and emerging technologies. For example, will digitized voice mail messages saved by VoIP systems fall under the same rules that govern the retention of e-mail and instant messaging (IM)? With the growing popularity of VoIP, it's a good bet that this technology will have an impact on regulatory compliance at some time. Mobile devices also pose some unique challenges, such as device-to-device messaging that skirts the corporate e-mail or IM systems.
But a more immediate concern is turning a compliance plan into action. In addition to expenditures for additional storage, companies should expect other costs such as training staff to help ease the impact of compliance. In some cases, compliance will expand storage operations to the point where additional staffing is required.
Harvard Medical School's Halamka created the storage manager position to help facilitate compliance activities and to advance the organization's ILM implementation. Halamka says the new position is just part of the repositioning of storage as a strategic asset to the organization.
At Citigroup, the extensive compliance effort required adding staff. "Your existing staff is going to have to devote more time to documentation, compliance training, certification and audit," says Shaun Mahoney.
For companies just starting down the regulatory compliance path, it's important for IT--and particularly storage managers--to step up and take a lead position. "IT people have to realize that they play a very important role in enabling compliance," says ESG's Gerr. But he adds that "IT managers have to change [their] frame of mind from a box-centric or systems-centric to an information-centric perspective."