After being bumped from the top backup and recovery software spot last year by BakBone Software, CommVault's Galaxy returns to the winner's circle.
Based on user comments from this year's Diogenes Labs–Storage magazine Quality Awards survey for backup and recovery software, most backup software vendors should expect an earful of complaints the next time they visit their customers. The following was typical: "The way they manage and update software licenses on their Web site is a complete joke. Whomever [sic] created that process should be fired after they spend a couple of months on the help line." Other comments weren't so polite.
Licensing aside, the very nature of backup contributes to the unsettling feeling some users have about their backup applications because the apps extend to all parts of a storage infrastructure. "Backup products tend to tell you what's wrong with your environment," says Brian Daniels, lead storage area network/Unix engineer at Catholic Health East in Newtown Square, PA.
But not all user experiences are so dire. CommVault's Galaxy (now part of the company's newly named Simpana suite), this year's enterprise-class winner, proved that some users are quite happy with their backup and recovery (B/R) software. "Galaxy is easily the most scalable product I have ever used and my product of choice," notes one user who requested anonymity because his company discourages employees from talking to the press. "After using this, I would be hard-pressed to ever look at TSM [IBM Corp. Tivoli Storage Manager], [EMC. Corp.] NetWorker or [Symantec Corp.] Veritas [NetBackup] in my daily operations." CommVault posted a 6.16 overall score (on a scale of 1.0 to 8.0), the highest score tallied by any vendor in the three years we've conducted B/R surveys (see the "Overall ranking" section of the "Rankings of enterprise backup and recovery software," below). This was CommVault's second time in the enterprise-class winner's circle. Our 2006 winner, BakBone Software Inc.'s NetVault:Backup, didn't generate enough user responses to be statistically valid this year.
In the small- to medium-sized business (SMB) category, EMC Retrospect returned to the top spot after a one-year hiatus. While Symantec's Veritas Backup Exec edged out Retrospect a year ago in a near-dead heat, Retrospect had a bit more breathing room this year; its 5.29 score topped Backup Exec's 4.93 handily (see the "Overall ranking" section of the "Rankings of SMB backup and recovery software," below).
Retrospect garnered many positive comments, especially regarding its support for Apple Inc. Mac systems. "It supports Mac clients, Linux clients and Windows; it's very unusual to get that range of support," says Maurice Volaski, systems administrator at Albert Einstein College of Medicine in the Bronx, NY. Volaski has used Retrospect for more than 15 years and is a big fan of the product. "In terms of features, it's the best you can possibly imagine," he says.
Product scores improved this year vs. 2006. Hewlett-Packard (HP) Co. OpenView Storage Data Protector scored a solid second place among enterprise products with a 6.07, the only other product to have an overall rating that exceeded 6.0. Syncsort Inc. Backup Express, which scored a below-neutral 4.37 in 2005 and was unrated last year, jumped to a fourth-place finish this year with a respectable 5.23.
In the SMB category, CA ARCserve Backup (formerly BrightStor ARCserve Backup) finished third for the second year in a row, but improved significantly from a 3.93 in 2006 to a 4.25 this year.
In the sales-force competence category, CommVault and HP shared the enterprise top spot with each receiving a score of 6.07 (see the "Sales-force competence" section of the "Rankings of enterprise backup and recovery software," PDF). CommVault received its best score (6.35) in this category for the statement "My sales support team is knowledgeable." HP's best score (6.29) was for "My sales rep is easy to negotiate with." Both vendors received their lowest scores (a 5.75 for CommVault and a 5.94 for HP) for the statement "The vendor's licensing formula offers good value."
"The licensing scheme for [HP] Data Protector makes it extraordinarily attractive for us," says Mark Stewart, a storage administrator for the Air Force Personnel Center at Randolph AFB in San Antonio, "because we can add hosts and clients on the fly without being forced to procure additional licenses." That kind of flexibility gives Stewart some latitude in overseeing data protection for more than 350TB of mostly HP storage. "I can do whatever I want to in the environment and not be held hostage to 'Do I have licenses to activate this?'" adds Stewart.
Still, licensing received the greatest number of complaints in this survey, especially regarding prices and administration--Symantec's Veritas NetBackup was particularly singled out--and little has changed since our 2005 survey. In that year's article, we quoted Karl Lewis of the University of Michigan in Ann Arbor as saying Veritas' licensing was his "No. 1 complaint." Symantec seems to have been unable to develop a formula that users like. NetBackup received the lowest score of any product in this regard with a 4.11. EMC Retrospect had the highest score (5.20) for this statement among SMB products.
Like CommVault Galaxy, EMC Retrospect received its highest score (5.45) in the sales-force category for the statement "My sales support team is knowledgeable." This compares to Backup Exec's 4.88 on the same statement, which was also its highest score. Backup Exec didn't score 5.0 or above on any statement in this category.
Whether in enterprise or SMB firms, B/R software needs to address a variety of situations, including file system, database and NAS backup done locally and remotely. Therefore, we asked respondents to rate the products on the statement "This product is a complete solution." CommVault led the field regarding this statement with a 6.83 followed by HP with a 6.14. CommVault, which won the category with a 6.51 (see the "Product features" section of the "Rankings of enterprise backup and recovery software," PDF), had its highest rating (6.85) for "This product's file-system backup features meet my needs," while HP received a 6.62.
Interoperability has always been an issue, and is becoming more of a factor as companies add more data protection capabilities to buttress their backup apps. The statement "This product is interoperable with other vendor's products" typically garners some of the lowest ratings in this category, and this year is no exception. "Interoperability becomes challenging because you don't want to put all of your eggs into one basket," says Catholic Health East's Daniels. He likes how EMC NetWorker works with other EMC products, but is leery of lock-in. "It seems that once you get into one EMC product it's very difficult to get interoperability with anything else," he notes.
Backup reporting is also growing in importance as capacities grow and environments become more complex. Although advanced reporting tools are available from third parties, there's a fair amount of frustration regarding those capabilities in core backup apps. The Air Force Personnel Center's Stewart, who's otherwise very pleased with HP Data Protector, notes its reporting deficiencies. "I have no way of trending media utilization, errors within libraries," he says. "I would really like to have some hard, fast numbers to provide to management and something that I can look [at] to think about what's next."
SMB products, as a group, rated much lower than their enterprise counterparts. One would expect such products to have fewer features, but our survey normalizes for that discrepancy by asking how well a product meets the user's needs, not a line-item comparison of features. We speculate that some users are hoping to get enterprise-class capabilities at an SMB price and are ultimately less satisfied with a less-capable product.
Regardless of the reasons, Veritas Backup Exec received a 5.07 as a complete solution, best among SMB products vs. Retrospect's 5.00 and ARCserve's 4.26. Retrospect edged out Veritas Backup Exec to win the category with 4.93 vs. a 4.91 (see the "Product features" section of the "Rankings of SMB backup and recovery software," PDF). While users apparently like Backup Exec's file-system backup capability best, giving it a 5.38, it still placed second to Retrospect (5.48).
"In terms of getting the backup done, getting the restores done, Retrospect definitely comes through, no question," says Volaski at Albert Einstein College of Medicine.
Initial product quality
B/R is the single most labor-intensive task in storage management. Failed jobs, missed backups and the like require almost daily intervention by administrators. We asked respondents to rate the products on the statement "This product requires little daily intervention." CommVault was tops among all products with a 6.21, followed by HP with a 6.11. Retrospect and Backup Exec tied with a rating of 5.05 each.
Initial product quality, however, was the one category where HP Data Protector surpassed CommVault Galaxy. HP did so convincingly, with a 6.30 vs. a 6.09 for CommVault (see the "Initial product quality" section of the "Rankings of enterprise backup and recovery software," PDF). Aspects that HP users liked most included "This product uses tape efficiently," (6.69) and "This product delivers good value for the money" (6.68). CommVault users gave it a 6.32 for the "value" statement, while the SMB product delivering the best value was Retrospect (5.73).
Getting up and running fast is another key factor. "If you have any background, any experience with any other enterprise-class backup application, there's nothing within HP Data Protector that's going to throw you off," says the Air Force Personnel Center's Stewart. "The time it takes to become productive with it, even for a novice, is very short."
Retrospect also received high marks for "This product is easy to install" (6.67), its highest score in the category. Veritas Backup Exec was almost a point lower with a 5.71. Retrospect also topped the SMB group for the statement "This product is easy to use" with a 6.10 vs. Veritas Backup Exec's 5.41 and ARCserve's 4.81. "In terms of configuration, it's very trivial to do," says Volaski about Retrospect.
Although backup and recovery products are labor-intensive, IT organizations expect them to operate flawlessly. We therefore asked respondents to give us their ratings on the statement "I can operate this product in a lights-out mode." Scores for all products were low on this statement relative to the category average. CommVault's 5.78 was the highest of all the products, but substantially lower than its 6.11 category average (see the "Product reliability" section of the "Rankings of enterprise backup and recovery software," PDF). On our scale of 1.0 to 8.0, a 4.5 is neutral. Scores below 4.5 are fairly rare and indicate that users have a negative perception of the product. Three of the nine B/R product finalists were rated negatively on the lights-out issue: CA ARCserve Backup, EMC NetWorker and Syncsort Backup Express.
Dealing with errors is another common area of frustration for B/R administrators. When we presented the statement "This product's error handling is easy and intuitive," the results were even lower than the lights-out issue. Here, five products were rated negatively: CA ARCserve Backup, EMC Retrospect, EMC NetWorker, Syncsort Backup Express and Symantec Veritas NetBackup. The top three products were CommVault Galaxy (5.63), HP Data Protector (5.27) and IBM TSM (4.79).
On a more positive note, vendors generally received good scores with regards to platform support. CommVault had the highest single statement score (6.48) in this category for "This product's OS platform support is adequate" followed by TSM's 6.41. Users of most products also responded favorably to "The product meets my service-level requirements."
The only vendor to post a score higher than 6.0 for technical support was CommVault with a 6.03 (see the "Technical support" section of the "Rankings of enterprise backup and recovery software," PDF). Its highest score in this category was for the statement "The vendor provides support as contractually specified" (6.71). However, most other vendors also did well on this statement. The top score among SMB products in this area was Retrospect's 5.57.
Much of the user frustration about support is related to having problems properly escalated and getting support personnel equipped to deal with the issues. Daniels at Catholic Health East says it could be a hit-or-miss situation when he has a routine problem with EMC NetWorker, but bigger problems are a different story. "When we have really, really major issues they're really on point, on top of everything," he says, "and they're really quick to resolve the issue."
Being selective can also help you avoid support frustration. "There's a short list--a handful--of support engineers I know by name that I now ask for," says the Air Force Personnel Center's Stewart. "And if they're working on another case, I will wait for them to get free."
Support scores generally dipped for the statement "Support issues rarely require escalation," as well as for its companion statement "Support issues are resolved in a timely manner." Among the enterprise products, CommVault posted scores of 5.80 and 5.86, respectively. For SMB products, EMC Retrospect showed best in this area with scores of 5.21 and 5.43, respectively.
Our surveys also ask respondents if, all things considered, they'd make the same purchase decision today. In 2006, 67% of B/R respondents said they would. This percent dropped minimally to 66% in 2007.
Often, we find no correlation between these scores and the overall order of finish. However, this year there was a close correlation. Among CommVault users, 85.2% said they'd buy Galaxy again (see the "Would you buy from this vendor again?" section of the "Rankings of enterprise backup and recovery software," PDF). For HP, 84.2% of Data Protector users would buy again. The SMB group was led by Backup Exec's 55% positive response, which was very closely followed by Retrospect's 54.5% (see the "Would you buy from this vendor again?" section of the "Rankings of SMB backup and recovery software," PDF).
Given the trend toward disk-based backup, we also asked respondents to tell us if they use the product for such a function. Among TSM users, 85.7% use the product to back up to disk, while 79.6% of CommVault users do so. Among SMB products, 68.2% of Retrospect users do so, along with 64.5% of Veritas Backup Exec users.
Based on our 2007 survey, we conclude that users are generally satisfied with the features and functions of their B/R products. Common areas of complaint are licensing complexity and error handling. Neither aspect requires fundamental engineering modifications, so we hope vendors focus on them in the coming year. Perhaps then B/R products can escape the stigma of being the product that users "love to hate and can't live without."
Products included in the survey
Enterprise products: Asigra Inc. Televaulting*, BakBone Software Inc. NetVault: Backup*, CommVault Galaxy, EMC Corp. NetWorker, Hewlett-Packard Co. OpenView Storage Data Protector, IBM Corp. Tivoli Storage Manager, Syncsort Inc. Backup Express, Symantec Corp. Veritas NetBackup
SMB products: Arkeia Software Network Backup*, Atempo Inc. Time Navigator*, BridgeHead Software Inc. HT Backup*, CA ARCserve Backup, EMC Corp. Retrospect, Symantec Corp. Veritas Backup Exec, Yosemite Technologies Inc. Yosemite Backup*
*Didn't receive a sufficient number of responses to be included in the survey results
| About the Quality Awards: The Diogenes Labs–Storage magazine Quality Award III for backup and recovery software is the second installment in the third annual series of survey-based service and reliability awards. The Quality Awards are designed to identify and recognize products that have proven their quality and reliability in actual use. The results are derived from a survey of qualified Storage readers that assesses products in five main categories: sales-force competence, product features, initial product quality, product reliability and technical support. Our methodology incorporates statistically valid polling that eliminates market share as a factor. Indeed, our objective is to identify the most reliable products on the market regardless of vendor name, reputation or size.
This year's survey generated 824 valid system evaluations from 624 respondents, representing a 21% jump in system evaluations and a 25% increase in respondents vs. last year. The margin of error was 4% with a 95% confidence factor. Our survey included a list of 15 products, but only nine had a statistically valid number or responses. The list of finalists was similar to last year's with the addition of Syncsort Inc. and the deletion of BakBone Software Inc. Symantec Corp. Veritas Backup Exec had the largest number of evaluations (208) followed by Veritas NetBackup (152) and IBM Corp. Tivoli Storage Manager (132). Ninety-two percent of respondents had one or two backup applications, and only 18 respondents (2.8%) had four or more.