Organizations need to migrate data more often than they think. It's no longer just once every three years when the lease expires on the old storage and a new storage array arrives. Today, companies can find reasons to migrate data almost every week.
"Data migration has changed," says Greg Schulz, senior analyst, Evaluator Group Inc., Greenwood Village, CO. "It used to be used for technology replacement--now it's used for a wide variety of maintenance tasks." Even technology replacement has become a continuous event. "We now replace one-third of our disk storage every year," says Robert Bellanti, vice president of data center engineering at KeyCorp, a Cleveland-based bank. The bank's storage is still on a three-year lease, but one-third of it comes up for renewal every year. KeyCorp may also bring in additional storage to accommodate growth. As a result, the bank finds itself migrating data among storage arrays throughout the year.
Data migration for the purpose of technology replacement is just one of the reasons compelling organizations to engage in the process. Other reasons include disaster recovery, balancing workloads, and streamlining application development and testing. Data migration, which began as an occasional chore, has become part of the IT department's ongoing repertoire.
Even though it's becoming more common, data migration isn't simple. Complicating the challenge of moving large volumes of data from one system to another--and ensuring that the data arrives in usable condition--is the advent of 24/7 operations. The migration team no longer has the luxury of shutting down systems, moving data and then checking to make sure everything is right. The trick now is migrating data without disrupting the systems that rely on the data. "We're talking about our production environment," says Bellanti. "We can't just shut it down."
Whether the organization is looking to improve performance, merge data centers, simply refresh storage gear or whatever the reason, a data migration is a complex operation, notes Richard Harsell, engagement partner at Framingham, MA-based GlassHouse Technologies Inc., in Data migration: Preparation equals success. Among the difficulties he outlines is the need to thoroughly plan the migration; documenting the data, LUNs, volumes, as well as the old and new storage configurations; creating the appropriate volume groups and volume file systems; identifying and understanding firmware changes; testing the migration plan; and scheduling the migration.
Compounding the problem is the shortage of migration tools, especially vendor-neutral tools. Many companies rely on native copy commands or mirroring capabilities built into the storage platform. Others rely on standard backup and restore tools. The advantage of these tools is that they don't require you to purchase additional software or learn new skills.
But these tools have limitations. Tools provided by storage array vendors may not work with arrays from other vendors. Copying and mirroring utilities may degrade or even disrupt app performance during the replication process, or the firm may not have a sufficient window to run a backup and restore operation.
As a result, some organizations are turning to third-party tools that are independent of the storage arrays. These may be network-based tools like RealTime Data Migrator from FalconStor Software Inc., or host-based software from vendors like Softek Storage Solutions Corp. and Veritas Software Corp. (now merged with Symantec Corp.), says the Evaluator Group's Schulz. In other cases, adds Schulz, organizations may opt for third-party, disk-to-disk backup tools such as those offered by Innovation Data Processing (IDP), which can also be used for data migration, although they're not intended as such.
Host-based migration toolKeyCorp migrated its data the old-fashioned way--by shutting down systems and performing data set moves--for years. However, as the financial services industry moved inexorably into 24/7 operations, the opportunities to shut down the systems for data migration became scarce. "You can't do data set moves when you can't shut down the system," says the firm's Bellanti. Four years ago, the company turned to Softek's Transparent Data Migration Facility (TDMF), host-based migration software that could handle the migration without disrupting the applications.
The company manages 75TB of storage with one-third of that coming off lease each year. "We have a ton of data to migrate every year," Bellanti notes. The production environment alone uses 20TB of storage, which is mirrored to a disaster recovery site.
Initially, TDMF was used solely for migrating data from the storage going off lease to the new storage coming in. Once the IT group had the migration tool, it found another use for it. "We use it to rebalance volumes within a mirror," says Bellanti. This rebalancing is necessary when much larger capacity storage replaces old storage. With TDMF, the company can migrate 1.7TB of data to rebalance a mirror over a weekend.
A major Midwest-based retailer had a similar experience. It initially opted for TDMF to migrate data to new storage when older storage came off lease. "We would have thousands of mainframe disk volumes that we would migrate over a period of two to three months," says Mike Ritz, senior architect for the retailer's IT services group. The company currently manages 45TB of storage.
Then the retailer noticed hot spots developing when some database tables and their corresponding LUNs were repeatedly hit. "Using TDMF, we could move selected volumes and balance out the hot spots," says Ritz.
In other cases, the retailer added more storage due to growth. "Every now and then we would have to add another terabyte of disk and shift some volumes over to the new storage," Ritz explains. Again, they'd turn to the data migration tool.
Now that the company has the migration tool, Ritz has a short list of things to do with it. For example, most OS utilities let you move storage between like systems. But Ritz's firm often finds itself needing to move data between dissimilar systems, such as going from a 3GB LUN to new storage allowing for a 27GB LUN. "If you do volume replication, you're stuck with the same size. If you use a tool like TDMF, you can make changes [in LUN size] and the tool will work it out for you," he says.
In the future, Ritz may consider using the migration tool when testing new development. "In that case, we would migrate production volumes and make them available for a test," he says.
Network-based migration toolThompson Hine LLP is a Cleveland-based law firm with seven offices. It turned to FalconStor for network-based migration and replication following a failure that brought down its headquarters' SAN. They had intended to use a host-based backup product, but the vendor was unresponsive although disaster recovery seemed like a suitable use of the product. "We originally wanted to use it to move data from Cleveland to Cincinnati," says Gary Theus, field support supervisor.
The company installed the FalconStor appliance and began a daily replication of 120GB to 170GB of e-mail between the two offices. FalconStor's ability to identify only data that has changed, allowed the company to move just 10% of its daily volume by sending only the changes. The replication proved so effective that the firm rolled out FalconStor appliances to all of its offices and set up T1 links between them.
With the replication appliances in place, Thompson Hine gained confidence in its ability to move data efficiently and reliably. "Now that we can move data, we've got a number of plans," says Theus.
First up was an upgrade of its primary storage, a Hitachi Data Systems (HDS) array. "With FalconStor, migrating the data turned out to be a piece of cake," notes Theus. They set up a synchronous mirror to their secondary storage, a Nexsan Technologies ATABeast. At one point, they turned off the mirror and made the Nexsan their primary storage. "We rolled in the new HDS and connected it to the fabric," he adds. Then it was a simple process of mirroring LUN by LUN back to the new HDS from the Nexsan. HDS carved out the LUNs, and FalconStor was used to virtualize the LUNs. Given that the new storage used much larger disks, FalconStor let the team resize the LUNs and mirrors.
Multiple uses for migrationThe primary use for data migration tools is moving to a new storage platform, says the Evaluator Group's Schulz. However, he identifies other uses:
- Tiered storage -- combined with a management tool like AppIQ Inc.'s StorageAuthority Suite, the firm can identify data based on usage rules and move it to other storage tiers
- Disaster recovery -- move data to another platform for safekeeping
- Load balancing -- redistribute data for improved performance
- Prestaging data -- move data closer to where it will be needed; this is particularly useful with software downloads and fixed content
- Application development -- move a sample of production data to separate storage for use in testing
- Data warehousing -- move data from multiple sources and locations to the data warehouse
- Collaborative computing -- move large files, design renderings and other documents into a centrally accessible storage pool
There are other ways to do these tasks--extract, transform and load tools, groupware or business intelligence tools--but a flexible data migration tool can also do them. Once you get the right tool for one task, it's easy to find other uses for it.