Data migration: Preparation equals success

Moving data from one array to another is becoming more commonplace for reasons such as implementing new or upgraded systems, tiering storage or archiving older data. Data migration is a complex and painful process, but these steps can help ease the pain.

Data migrations are becoming a more frequent part of a storage administrator's workday. A data migration is a complex operation, whether undertaken to improve performance, merge data centers or simply to refresh storage gear. And because they're so complicated, many migrations run into issues that delay or stop the process. The following steps will help you to avoid hidden issues and prepare for a successful migration.

Most migration projects kick off with a feasibility study to determine if the project will reduce cost, risk or improve the user experience. At this stage, the project's concepts are articulated to relevant staff members and, in some organizations, the project is officially sanctioned at this time.

The planning process is next. During this time, the project team determines the best approach for the migration and addresses any risks. At this stage--or earlier if possible--it's important to identify the appropriate level of sponsorship for the data migration. The right level of sponsorship can also help to "grease the skids" for setting up meetings and getting responses to requests. A director from the business side is usually the most effective sponsor to help alleviate the typical issues that arise between IT and business groups. A strong and simple communication plan is paramount to a smooth migration project.

Checklist of migration activities

Scheduling the migration
While there are several ways to arrange, coordinate and schedule a migration, minimizing the impact on business users should be a primary focus. Scheduling the migration based on business needs rather than on the IT department's desires should be factored into other facets of the migration such as the manpower required for the project.

It's easier to arrange the migration based on the server OS or the volume of data to be migrated. Because there's often little control or process around how apps are portioned out, you may be required to inventory the apps at the host level. This can be a taxing project, so allow enough time to obtain the information and vet it with appropriate business/system users. If the inventory process is painful, it's probably a good time to implement a more effective configuration/change management process. For a data migration effort, the following attributes of a server/application inventory should be included in a list of all affected or source servers:

  • OS and version/patch level of server
  • Target host (if changing) with OS and version/ patch information
  • Current array and associated hardware
  • Volume information (allocated, used and free)
  • LUN information
  • Target array and associated hardware
  • User information, including apps/databases, department, department contacts, DBA contacts and end users (especially if a file system with user data is involved)

While not absolutely necessary, other items that might be included include cabling requirements, server IDs, free slots and physical locations of the servers. Don't be surprised if someone from a business unit identifies incorrect information, especially if the business/IT liaison acts as an application administrator. It's difficult for IT to keep an inventory that's 100% correct, so it's good to periodically review the inventory. A final consideration is the criticality of the data to be migrated. Again, the project manager will need to ensure that the business units are the ones categorizing the importance of the data (based primarily on the importance of the application).

The volume of data
Once the data has been logically grouped by business unit and importance, the volume of data to be migrated should be considered to refine the migration schedule. Volume isn't only the amount of data; it's also the number of files. Block-level copying of data can take longer if there are many small files, as is often the case with unstructured data. Of course, there isn't much flexibility regarding the time allotted for the migration, assuming it can take place only at night or during a weekend. Storage admins can use their empirical data to estimate the time needed to migrate a certain amount of data from Platform X to Platform Y. Each platform has its limits, so consult with the appropriate source and target vendors to gather this information. The transfer rate should also be verified during premigration testing.

Knowing the transfer rate and the amount of data to be migrated will enable the project team to fine-tune the migration schedule. Even using an offline migration process, experience has shown that it can take up to twice as long as the estimated transfer rate. If mirroring is used as the migration method, user access will prolong the migration; users will also see performance degrade. If possible effects aren't communicated appropriately, unnecessary help desk calls or user frustration may result.

Typical data migration challenges

Help desk support
Arrange help desk support depending on the complexity of the migration. Provide help desk staffers with appropriate details so they can field calls related to the migration. For example, if the migration is between a Unix box and a Network Appliance box, path names will change and users will need to remap their shares. Another example would be migrating shares of Mac users. Macs will require special handing and, in some cases, special software, such as DAVE, to ensure transferred files will be readable. These types of variations or complexities should be clearly conveyed to the help desk.

Checklist and tools Migration and test plans should address:

  • Method of migration, such as mirroring or copying
  • Details of the premigration, migration and post-migration steps
  • Required testing scripts
  • Back-out plans

There are many tools that can help ease the migration process (see "Network-based migration tools" at and "Host-based migration tools" at Most data migrations are conducted using host-based tools in an offline/manual mode because this method reduces the risk of losing data. For the majority of migrations, native commands such as "ddcopy" and "xcopy," or the mirroring capabilities of the storage platforms, work well. Some migrations may even be performed using a backup and restore approach. These approaches have the advantage of not requiring any additional software, hardware or new skills.

Premigration steps
Once general planning is complete, it's necessary to determine if any upgrades or patches are required to support the migration. If you have a robust change control or configuration management program, this information may already be available. However, information should be reviewed to minimize the chance of glitches arising during the migration. Check the HBAs, switches, source and target platforms, and operating system to ensure that these are at the right version to support the migration and ongoing operations (see Checklist of migration activities).

Once the required updates have been determined, correct any items that need updating. Firmware versions, while good enough for operations, will often require updating to support the migration. Upgrades may require a system shutdown, so all business users should be notified so they can plan accordingly. It's also recommended to have a few days between the premigration shutdown and the migration to allow for troubleshooting. This waiting period helps systems to stabilize and provides you with the time to resolve any issues that arise as a result of the hardware/software upgrades before the actual migration. The waiting period will vary, but three to seven days is recommended.

The migration steps must address the appropriate level of detail based on the criticality of the data. In particular, attention must be paid to data controlled by one or more regulatory requirements. In some instances, documenting the migration at a high level will be sufficient. However, with the goal of minimizing risk, it's helpful to provide enough details (and then test against these details) to reduce the number of operational questions during the migration or the need for workarounds. It's difficult to definitively state the sets of detailed tasks because each environment is different; however, when unsure, add more detail.

Why migrate data?

Testing requirements
Migration testing should include both storage and applications. Storage-related testing ensures that the data was moved from Platform A to Platform B. If the migration is completed offline, the testing should include a comparison of pre- and postmigration volumes, number of files, number of database records and so on. Running a checksum is optional, but it could take a long time depending on the volume of data. If the migration is completed online, it's harder to measure these parameters. It's helpful to perform a mirroring migration in phases by dividing the volumes to be migrated into a series of smaller, sequential migrations.

App testing requirements will vary depending on the applications. For environments under regulatory control, a more detailed level of testing is probably required. The team should consult with the internal QA staff or compliance officer to determine the exact requirements. At a minimum, it's prudent to develop and document pre- and postmigration app tests. Running tests on the original and migrated data ensures the successful operation of the app prior to "turning off" the old storage. Some key testing milestones include:

  • Developing test scripts to run pre- and postmigration
  • Comparing pre- and postmigration test results
  • Verifying the ability of the app to read from the target storage
  • Verifying that pre- and postmigration reports or analyses are identical
  • Explaining any differences that might be expected

Back-out planning
To minimize downtime, adequate back-out planning is recommended. At each Go/No-go decision point, the team should determine if the migration can proceed and, if it can't, execute the back-out plan. Go/No-go points may occur at the completion of the migration and storage-related testing, at the end of app testing or other logical points in the migration plan. Depending on the migration method, back-out actions can include tasks like not disconnecting the old array, not exchanging mounts points, reactivating source volumes groups or restoring from tape if data becomes corrupt.

When the migration is complete and all apps have tested satisfactorily, disconnect the old array's cabling or break the mirror as soon as the postmigration testing is complete. If required, the storage array may be repurposed for other roles such as another tier of service or development. Finally, complete the paperwork for the migration and testing documentation, and obtain the necessary sign-offs. A postproject review can be helpful in identifying areas that can be improved to reduce risk, improve the user experience or reduce the cost of the migration effort.

Dig Deeper on Storage migration