We've come a long way in the decade since AWS Simple Storage Service debuted. No longer just for developers or...
buckets of objects, companies today deploy cloud storage for a variety of architectures (i.e., block and file as well as object) and uses, ranging from backup and disaster recovery to archiving and analytics. Even primary storage for workloads can now run in the cloud. With benefits such as increased agility, pay-as-you-go fees and near-infinite scalability, what more could you want?
Well, there is one thing. You may one day need to switch cloud data storage providers, and this can be a painful exercise, even under the best of circumstances.
Reasons to switch could include finding your current cloud storage provider's service levels and capabilities not up to snuff (e.g., to meet availability or performance levels required for apps or expanding uses).
For example, if you initially sign up for a cloud backup service, but later require cloud-enabled disaster recovery or analytics, too, you may discover the provider's service levels, functionality or geographic coverage doesn't support these additional cases. Second, those in industries with strict or rapidly evolving data security and compliance requirements will likely find some providers more adept than others at adapting and strengthening their offerings. Then there's the most likely reason of all: Your provider has gradually increased fees for storage access, data transfer and other services, reducing the cost-effectiveness of its service.
If you find cloud data storage providers that better satisfy on any of these scenarios, then you'll likely want to make the switch. Doing so isn't necessarily a walk in the park, however.
Qualify potential new providers
What's involved in moving data and service contracts? If you're storing large amounts of data in the cloud (hundreds of terabytes or even petabytes), the process can be challenging and time-consuming. Before embarking, it's important to ensure the new provider's services and capabilities accommodate your data storage needs. For example, when migrating unstructured data, check that it supports the file protocols or object storage access APIs you're using or planning to use. Do its service-level agreement (SLA) terms support your data availability requirements? How will overall costs compare? Take advantage of cost calculators and other resources from your current and potential new providers to determine how much more or less you'll be paying each month.
If you're also moving apps, check on the portability of the virtual machine (VM) images or app containers you're running, and decide whether they can simply be migrated or will need rebuilding. Also, qualify whether the new provider meets your application performance objectives and other critical requirements (e.g., for network latency or total throughput) by doing a small-scale trial or proof-of-concept. Though app migration is beyond the scope of this article, we bring it up because you'll need to include it when planning if you're currently running or intend to run workloads in the cloud.
Choose a migration approach
Once you're comfortable that a new provider can address your needs, start scoping your migration effort. You'll need to answer some basic questions:
- How much data will migrate?
- How long will it take given the options available?
- Will users require access to data during the migration? If yes, how will you ensure access while managing costs?
- What recourse or contingency plan can you implement should migration stall or the new provider's services and capabilities fall short of what you expected?
The answers determine how and when to migrate. Then, evaluate the available migration options of the new provider, which generally boil down to the following:
- Options should include online migration of encrypted data over the public internet using a "brute-force" approach such as Secure FTP (SSH-2 based) or similar technology.
- They should allow online transfers using virtual private network (VPN) tunneling via a secure private connection.
- They should provide transport of bulk data via physical media (e.g., portable storage devices).
- And they should include transfer via a provider appliance, third-party tool or service.
While most users choose to employ one of these methods, you can certainly use more than one approach if it makes sense.
The first approach works best when transferring relatively small data sets, since it's subject to the congestion and unpredictable latencies of the public internet and may consume lots of bandwidth. As a result, data transfers may take days or weeks to complete, even for a few terabytes. Remember, while data transfer costs may be reasonable, your current provider will still charge for outbound data transfer.
Calculate data migration costs
You will incur three categories of cost when migrating cloud data among cloud data storage providers:
Data transfer out (egress) costs. Based on the amount of data transferred out of your provider's site to the internet, major cloud data storage providers such as Amazon Web Services (AWS) and Azure currently charge between 7 cents and 9 cents per gigabyte for the first 150 TB of data moved, with the first 1 GB to 5 GB free. Each provider has discount tiers, and the price per gigabyte declines progressively as the amount of data crosses to the next tier. AWS also offers transfer acceleration for an additional 4 cents per gigabyte in cases where customers hasten transfers by routing through less congested edge locations.
Data transfer in (ingress) costs. No major provider charges for incoming data transfers and uploads, regardless of amount of data moved. We don't see this changing anytime soon, but you should confirm before initiating a transfer.
Costs of actually moving data. Whether transferring electronically or via physical means, you incur costs. For the former, the major cost will be based on bandwidth consumed, with a provider surcharge for port speed if transferring via VPN tunnelling. When moving data via HDD, you pay for the media and mail or courier services. You will incur a surcharge when using an appliance-based transfer service and, of course, pay separately for utilizing a third-party tool or service.
Major cloud data storage providers and selected independent software vendors offer online cost calculators or guidelines to help with estimating data transfer costs.
The second option lets you migrate rapidly, but can be costly when you migrate large amounts of data. This approach requires working with major public cloud data storage providers, such as AWS or Microsoft Azure, that have a relationship with the same network or data center services provider, such as Equinix. You would directly connect with the network services provider, which in turn can provide a VPN tunnel connection to your cloud storage accounts. Data would then flow securely from your existing cloud provider through the network services vendor and back out to the new provider's service.
The major advantage here is you can set up secure and reliable connections that aren't subject to the vagaries of the public internet for data transfer between providers. The major downside is the cost of the VPN tunnel services, which can be expensive. For example, AWS Direct Connect VPN tunnel service charges both an hourly charge per port (based on speed) and per-gigabyte egress fee, while Azure ExpressRoute service charges a monthly fee based on port speed. If you need to transfer critical data quickly, the costs could be worth it, but in most cases, it wouldn't be cost-effective.
Use the third alternative when planning to transfer large amounts of data to a new provider, but uploading or downloading over the network is unattractive due to limited bandwidth or high costs. You first transfer data from an existing cloud storage service, typically by sending the provider an HDD and creating an export job online specifying what data to transfer. The provider then moves your data, encrypts it and returns the HDD to you. You then ship it to your new provider for upload. Sometimes, you can streamline the transfer by specifying the HDD be shipped directly to the new provider. This process sounds inefficient, but it's generally reliable. Azure Import/Export and AWS Import/Export Disk are examples of transfer services that utilize physical media.
Cloud data storage providers clearly aren't motivated to ease data migration to other providers. As a result, you'll probably find the process more cumbersome and costly than expected. This has led a number of third-party vendors to introduce tools and services that facilitate migration efforts (see "Third parties that help with cloud migration"). In addition to these third-party options, some cloud providers also offer file sync and sharing or replication and recovery services that ease data transfer to servers in your data center, where files can, in turn, be migrated to a new cloud storage provider. When in doubt, talk to both your current and destination providers to understand other available tools and services that might help simplify or expedite your migration effort.
Third parties that help with cloud migration
For obvious reasons, major cloud data storage providers tend not to make it easy to export data to another service. A number of third-party vendors step in to fill the void, though.
Mover offers connectors among more than two dozen cloud providers, web services and databases, plus cloud-hosted migration services to help move files. SkySync enables file migration or synchronization between Microsoft Azure and AWS clouds or those public clouds and on-premises storage. Other vendors, such as Cloudyn, can help with monitoring and managing cloud files and workloads during migration. Also, check out storefronts like AWS and Azure's marketplaces for more tools that aid with migration.
Avoid cloud lock-in and ease migration
What can you do to avoid getting stuck with a cloud provider? We believe the following guidelines will help:
Vigorously qualify your short list of providers upfront to reduce the risk of having to switch later. As part of due diligence, determine whether they offer the cloud data uses you're interested in today or might expand to them in the future. Can they accommodate your requirements for availability, performance, security and other key SLA metrics? Carefully review the SLA terms and spend a month conducting a trial assessment. Are costs reasonable for the selected uses? Reach out to peers for their experiences with providers.
Consider how portable your cloud data and apps would be should you need to change providers. Will data be stored in a format that'll allow for easy movement down the road? Similarly, are the VM images or application containers you're running relatively portable? Most cloud data storage providers run customized hypervisors (based on Xen, KVM and so on), which means images may need to be modified before transfer. Converting to an open VM format (like open virtualization format) may be helpful, but the overall migration process will require some effort when running lots of VMs. Containers tend to be much more portable, but few enterprises have more than a handful of apps running in containers today.
Check out the provider's tools, services and associated costs for moving data out of the cloud. Would these enable you to cost effectively migrate assets to new providers? Are third-party tools and services available to fill any gaps? Think through the likely migration process, and determine how much IT staff time and other resources may need to be invested. Can you afford to take on a cloud-to-cloud migration project given the amount of data you'll be storing with your new provider?
If storing primary data is critical to your business in the cloud, consider how you might continue to provide access during migration. For example, you may want to stage the migration effort, replicating or accelerating the transfer of more important data to the new cloud, or do it off hours or at times of lower demand.
Tier data via a cloud storage gateway or similar platform to avoid getting stuck with critical business data in the cloud that's difficult to move. With this hybrid storage approach, primary and active (transaction-oriented) data is stored in your data center, while less active or inactive (secondary or tertiary) data may be stored in the cloud. Tiering can help avoid the unpleasant and costly prospect of migrating "warm" or "hot" data between providers.
About the author:
Jeff Byrne is a senior analyst and consultant at Taneja Group. He can be reached at email@example.com.
How data backup and archiving differ
A beginner's guide to cloud-based storage services
Compare, contrast cloud-based storage
Google adds new cloud data transfer service to compete with AWS