Published: 12 Jul 2007
The third year of the Diogenes Labs-Storage magazine Quality Awards for enterprise arrays saw some shifts among the vendors, but a familiar theme prevailed as Hitachi Data Systems won again.
In horse racing, it's called the Triple Crown. In bowling, it's known as a turkey and in hockey, it's a hat trick. We're inclined to call Hitachi Data Systems (HDS) Corp.'s three victories in a row in the Diogenes Labs–Storage magazine Quality Award for enterprise arrays "remarkable."
"Their hardware is rock solid," says Carlo Tosti, technical specialist at TD Bank Financial Group in Toronto. Tosti's group has a variety of HDS TagmaStore Universal Storage Platform (USP) and Lightning 9900 Series arrays in its shop, which over the years has evolved into an all-HDS environment. Keith Fisher, manager of information processing in the Office of Information Technology for the State of NJ, concurs with Tosti. At one point, the state agency had a mix of EMC Corp., HDS and IBM Corp. storage. "As the boxes got old, went away and we bought new stuff, it was all Hitachi," says Fisher.
Sun Microsystems Inc. rebounded nicely in the 2007 survey, rising from 2006's last-place finish (see "Quality Awards II: The elite of enterprise arrays," Storage, September 2006) to second this year. Sun's systems had scored well in 2005 (second), but Sun was the apparent victim of organizational upheaval following the StorageTek acquisition in 2006. Last year, EMC moved up to third from fifth place in 2005, and held onto third again this year. Interestingly, EMC is sandwiched between two HDS resellers: Sun and Hewlett-Packard (HP) Co., which finished fourth overall. IBM slipped from a fourth-place finish in the previous two surveys to fifth this year, but by the slimmest of margins (.01 points behind HP). Network Appliance (NetApp) Inc. was included in our enterprise array survey for the first time. NetApp has generally fared well in our NAS surveys, but its last-place finish among SAN arrays illustrates how brutally competitive the enterprise category can be. 3PAR was also included in our survey, but for a second year failed to garner enough responses to be statistically valid and couldn't be considered for the award.
'07 scores trump those of '06
Overall vendor scores rose across the board. Last year, scores ranged from Sun's 4.94 to HDS' 5.76, a spread of .82 points. (We use a 1.0-8.0 scoring scale, where 8.0 represents the best score.) This year, scores ranged from 5.70 (NetApp) to 6.31 (HDS) for a spread of just .61 points despite having six finalists instead of five (see "Overall rankings" PDF). HDS' year-over-year improvement of .55 was solid, but Sun's 1.18 was the biggest one-year jump. Even HP, which fell from second to fourth, saw its score rise from 5.54 to 5.93. If there's a silver lining for NetApp, it would be that its overall 5.70 would've been good enough for second place last year and represents very respectable customer satisfaction.
Survey responses also revealed which system could be considered the "heavy lifter" of the group. To determine this, we focused on the system with the highest percentage of respondents who indicated having more than 200TB of storage deployed on that vendor's product line. In this measurement, EMC was the hands-down winner; 30% of its respondents had deployments in excess of 200TB. HDS was second with approximately 20% of its respondents having more than 200TB, while NetApp had 15%; HP and IBM tied with 10% each.
In keeping with our established survey methodology, we asked readers to rate the vendors in five categories: sales-force competence, product features, initial product quality, product reliability and technical support. HDS won every category last year, but not this time. Three other vendors--EMC, Sun and IBM--had the high score in at least one category, either by dint of a tie or outright.
Although HDS topped the sales-force competence category (see "Sales-force competence" PDF), its 6.13 score tied for the lowest high score of any category. Within the category, its lowest score was for the statement "My sales rep keeps my interests foremost" (5.82)--the only statement in this category for which HDS scored below 6.0. But HDS wasn't alone; every vendor had its lowest category score for the same statement. NetApp scored 4.89 with all other vendors receiving a 5.20 or higher. Sun had the highest response (6.45) for "My sales support team is knowledgeable."
"They understand the business that we're in, the importance of the data--they understand our availability concerns," says Ken Craigie, global IT asset manager at GXS Inc., Gaithersburg, MD, when describing his firm's relationship with EMC. He added that negotiations between GXS and EMC are tough but effective because "you can tell a partner from a predator usually in the first couple of interactions."
We asked respondents if they purchased directly from the vendor or through a reseller. EMC had the highest percentage of direct sales (82.9%) and IBM had the lowest (56.2%), with all other vendors receiving between 60% and 69%. We've never found a correlation between percent of direct sales and order of finish. That's true with this survey, where there was no correlation between customer satisfaction and the sales force.
As we've noted, the enterprise array market is highly competitive. It's also relatively mature, so products have greater parity. In this year's survey, three vendors led the features category with identical 6.23 scores: EMC, HDS and Sun (see "Product features" PDF). While such a three-way tie is largely coincidental, it illustrates our point about product parity. The three were followed by HP (6.12), NetApp (5.80) and IBM (5.65).
But important features--such as replication--do sometimes rise to prominence. "The thing that I like the most is [HDS'] replication capability," says Tosti at TD Bank Financial Group. "TrueCopy--both synchronous and asynchronous--that's really fantastic." The State of NJ's Fisher notes that his agency uses a variety of HDS replication products and "we're very pleased with them." Craigie at GXS echoes those sentiments: "EMC is replicating our most important data far away from our primary data center where we can recover it and meet our clients' SLAs [service-level agreements]."
Although EMC, HDS and Sun tied in this category, how each arrived at that score was substantially different. HDS had the highest score (6.71) for a single statement, "This product scales to meet my needs." But HDS was the only one of the three to have two sub-6.0 scores in this category. Its lowest, a 5.82, was for "This product is interoperable with other vendor's products." EMC bested the field for "This product's mirroring features meet my needs" (6.66). It's only sub-6.0 mark (5.44) was for the interoperability statement. Sun was a study of consistency, having neither the highs nor the lows of the other two. It received two 6.48 scores for "Overall, this product's features meet my needs" and the scalability statement. Relative dissatisfaction with interoperability was a theme, though, as Sun received a 5.74 for this criterion, its lowest score in the category.
Initial product quality
Our initial quality section asks users to tell us how easy a system is to set up, configure and get running. Given that HDS and its OEM partners, Sun and HP, finished 1-2-3 in this category, it's fair to say HDS delivers a very favorable out-of-the-box experience (see "Initial product quality" PDF). "The process is very, very smooth," says Tosti at TD Bank Financial Group of HDS' installation procedures, which includes doing everything up to disk formatting.
HDS got its highest score in this category (6.62) for the statement "This product was installed without defects." HP (6.32) also scored well on that statement, but NetApp (6.24) snuck past Sun (6.05). NetApp, known for easy-to-use NAS appliances, scored well on "This product is easy to use" (6.19), but was second to HDS (6.39). IBM was the only vendor that didn't receive a 6.0 or higher for any statement in this category.
So which vendor offers the best value for the money in the opinion of our readers? Sun, by a whisker. Sun users gave the company a 6.41 for "This product delivers good value for the money," barely edging out HDS' 6.40. HP was a distant third (5.87), followed by EMC (5.70), NetApp (5.66) and IBM (5.54).
"We've never lost data," says Fisher about the State of NJ's HDS systems. "That's across everything, from the big enterprise boxes down to the AMS." HDS' highest category score (6.71) came in the product reliability section (see "Product reliability" PDF). Even so, EMC posted a 6.60, which was good for second place. Both HDS (7.04) and EMC (6.89) received their respective highest marks for "This product experiences very little downtime."
"They've been very reliable for us," says GXS' Craigie of the multiple EMC systems installed in the organization's U.S. and European locations. "Downtime is completely not acceptable at GXS for our primary array."
IBM joined HDS and EMC as the only vendors to score 6.0 or higher on all product reliability statements, with its highest score (6.36) for the statement "This product experiences very little downtime."
The statement "Patches can be applied nondisruptively" has proven to be a bellwether in previous surveys, and was again. We've never seen a vendor score poorly on this statement and perform well in the overall standings. HDS led the group on this statement with a 6.71 followed by EMC's 6.63. Sun scored a solid 6.52, HP a 6.26, IBM a 6.06 and NetApp trailed with a 5.28.
IBM proponents may be disappointed in the company's overall showing, but its top rating for technical support in 2007 is a nice consolation (see "Technical support" PDF). The company received its highest score (6.62) for "Vendor provides support as contractually specified." HDS had an even better 6.84 for this statement, but had its category average dragged down by a 5.22 on "Vendor provides adequate documentation and support materials." IBM also had the top score (6.27) for "Vendor takes ownership of the problem." In terms of the support group rated most knowledgeable, HDS' 6.35 nudged out IBM's 6.33.
Do it all over again?
In our surveys we ask users if, all things considered, they'd make the same purchase decision again. This 20/20 hindsight question is more interesting than a future buying-intention query, as momentum may cause users to continue purchasing a system even when less than thrilled with it. We noted an interesting non sequitur in this year's survey in that overall scores rose, but changes in willingness to repurchase were all over the map (see "Would you buy from this vendor again?" PDF). This year, 84.6% of HDS customers would do it all again--lower than last year's 88.4% and the biggest drop of any vendor. Sun rose almost 14 percentage points to 82.6%. NetApp was third with a healthy 77.8%, indicating that respondents are either willing to work through NetApp's growing pains in this category or feel that, on balance, NetApp is a favorable provider. EMC's rating dropped slightly from a 79.3% to 76.8%. HP climbed slightly to 71.4% from 69.8%, and IBM rose from 62.2% to 68.1%.
Many respondents evaluated more than one vendor's system, so we also compared the ratings for multiple vendors as submitted by the same respondent. Comparing head-to-head win rates, Sun led with a 44.4% win rate. Again defying the order of finish, NetApp was second with 41.2%. HDS was third, winning one in three comparisons. Does HDS' declining rate of willingness to repurchase and head-to-head win rates suggest chinks in its armor? Stay tuned for the 2008 results.
Products included in the survey
The following products were included in the Diogenes Labs-Storage magazine Quality Award Enterprise Arrays III survey.
EMC Corp. Symmetrix 3000 Series/8000 Series/DMX Series/DMX-3 Series, Hewlett-Packard (HP) Co. StorageWorks XP Series, Hitachi Data Systems (HDS) Corp. Lightning 99xx Series or TagmaStore Universal Storage Platform (USP) Series, IBM Corp. ESS TotalStorage ("Shark") F Series/800 Series/800 Turbo Series/DS8000 Series, Network Appliance (NetApp) Inc. FAS6000 Series, Sun Microsystems Inc. StorageTek 99xx Series, 3PAR InServ Storage Server S400 or S800*
*Didn't receive a sufficient number of responses to be included in the survey results.
About the survey
The Diogenes Labs–Storage magazine Quality Awards are designed to identify and recognize products that have proven their quality and reliability in actual use. The results are derived from a survey of qualified Storage readers who assess products in five main categories: sales-force competence, product features, initial product quality, product reliability and technical support. Our methodology incorporates statistically valid polling that eliminates market share as a factor. Indeed, our objective is to identify the most reliable products on the market regardless of vendor name, reputation or size. Products were rated on a 1.0-8.0 scale, where 8.0 is the most favorable possible score.
This year's survey yielded 408 valid respondents on 687 systems for a margin of error of +/- 5% with a 95% confidence factor. The single largest group of respondents was for EMC Corp. (142); Hitachi Data Systems (HDS) Corp. had 78, followed by IBM Corp. (72). Sun Microsystems Inc. had the smallest number among award qualifiers (23). We allow respondents to rate families of systems, not just current models. Network Appliance Inc. is the only vendor without three-plus year-old enterprise arrays in the field. EMC DMX-3 was the most numerous reported system in the field (98).