adam121 - Fotolia


Cloud file services: Benchmarking considerations

Cloud storage can save costs, but choosing the right service is vital. This tip covers important considerations when planning a cloud file services benchmarking project.

Storing company data in cloud file services is a viable option for businesses of all sizes. And with cloud storage costs steadily decreasing, there is often a strong economic argument for moving data to a secure, off-site location.

Once key data assets are migrated to the cloud, you need a cloud file service to perform for your users. However, not all cloud services are created equal. Fortunately, you can do some basic benchmarking of your own to identify potential performance deficiencies with prospective services before you commit.

Cloud file services consideration No. 1: Internet environment

Perhaps the biggest variable in the performance of your cloud file service is the Internet connection between you and your provider. While the upload and download speeds of your connections serve as your maximum possible throughput levels, actual throughput can vary based on the other equipment that might be sharing your internal network, as well as the Internet bandwidth between you and your cloud file services provider.

It's essential to run your benchmark from a wired Ethernet station and not a wireless LAN (WLAN) connection. You don't want the performance of the WLAN, which can vary depending upon load, to impact your cloud performance measurements.

To start, run a speed test to benchmark and reaffirm the maximum upload and download throughput of your Internet connection. You can use any speed test site -- it doesn't have to be one maintained by your provider. I find AT&T's speed test easy to use. Rerun the test and note the results each time to reconfirm the maximum bandwidth you have available.

Some cloud file service providers have proprietary speed tests that can provide "best case" benchmarks of throughput from your facility to their data center. Unfortunately, these are likely for internal use only. However, if you are a big enough company, some arm-twisting may get you access to these results so you know the practical limits of your throughput all the way to the cloud data center.

Finally, since the traffic from your location to the cloud file data center might vary, you should plan to run tests during different test windows and note when each test was run.

Cloud file services consideration No. 2: What to test

Determining what to test -- the most important part of the process -- is the easiest to lose control of. Newbie testers typically take an expansive approach, which only works if you are a government or research organization with a big budget. You may initially say, "Let's test three different OSes, using three different browsers with five different file sizes, and upload and download at three different times of the day." It would be nice to collect that amount of data, but a benchmarking scenario of that scale could approach 3,000 different tests -- and that's with a single service and without evaluating mobile versions of your applications.

The goal is to get all the critical information you need from as few tests as possible. A good way to accomplish this is to include a prototype phase. That is, once you have a list of elements that you believe would be useful variables in your testing, run some informal tests to see if changing those variables will likely enhance testing. Does tripling your testing load by running morning, midday and afternoon tests show any differences in your environment? Run some quick tests to find out. If the differences are minimal, it might not be worth the effort.

The sync program is the heart of cloud file services. It runs in the background and moves data between client and cloud. This function should be the focus of your testing.

Your first decision will be the OS environments to test. Microsoft Windows and Apple OS X will likely top your list. Fortunately, most cloud file sync services have a single version for Windows and a single version for Apple OS X, so you won't have to test multiple versions of each OS.

Many cloud file services also have versions for mobile devices running Apple iOS and Android and perhaps other mobile environments. My advice is simply to note the platforms available and save any feature or performance testing on the mobile platform until you finish your primary testing. Then you can test your preferred vendor to ensure the mobile platform also meets your requirements.

The sync program is the heart of cloud file services. It runs in the background and moves data between client and cloud. This function should be the focus of your testing.

While each service is different, you can designate a folder for sync between cloud and client. By default, the synchronization takes place in real time. While that is the way you will use it normally, you do not want that to happen in your test. When you test, you need to know exactly when the sync starts and when it ends. Determining these two points can be your biggest challenges.

Services differ in how they implement turning sync on and off, so you will need to find out how to do this. When you're ready to run a test, load your folder with the test data -- that folder can reside on your client or in the cloud, depending upon whether you are testing upload or download -- and manually turn on the sync. Make sure to note the exact time this was done.

The most important factor will be the data you choose to place in your test folder. Tests by my firm have shown that not only is there overhead per byte transferred, but per file. To exercise both functions, you should test synchronization -- separately -- of a few large files and many small files.

Our tests to date have not shown much effective relative performance difference between synching a 100 MB file and a 2 GB file, for example. So pick a large file size that is relevant for your environment. Unless your workplace does a lot of video work, it is unlikely you will have a lot of multi-gigabyte files moving around the cloud.

With our testing, we evaluated per-file overhead by synchronizing a directory containing several thousand small files, approximately 10 KB in size. We found that where one service could process 15 files per second, another could only process one file over two seconds. When you need to move thousands of files using cloud file services, this can make an enormous difference in how long it takes the synchronization to complete.

In any case, be sure your choice of data for testing reflects the characteristics of actual data that will flow across the service. This is the only way to make a realistic evaluation of how cloud file services will perform in your environment.

Now that you've completed your cloud file services testing project, learn how to measure and interpret your benchmarking results.

Next Steps

Alternatives to cloud file sync-and-share services

How cloud file sync differs from cloud storage

Cloud file-sharing pilot mitigates risk

Dig Deeper on Unstructured data storage