Quality Awards II: The elite of enterprise arrays

Hitachi Data Systems consolidates its claim as the top-rated enterprise array by sweeping all five user-experience categories in the 2006 edition of our survey.

This article can also be found in the Premium Editorial Download: Storage magazine: Upgrade path bumpy for major backup app:

Hitachi Data Systems repeats as winner of the second edition of the Diogenes Labs-Storage magazine Quality Awards...

for enterprise arrays.

Nothing symbolizes the prestige of storage in a data center more than enterprise-class disk arrays. Discussions about enterprise arrays are always more emotional and intense than other storage product categories, and justifiably so. That emotion is driven by the reality that "Tier-1" enterprise storage holds the data that fuels the organization's most demanding applications. Mission-critical application failure can cost businesses millions of dollars in seconds. Although vendors talk in terms of "nines of service," in practice, IT organizations are upset if their enterprise arrays ever fail.

Products in our survey
The following vendors and products were included in the 2006 Diogenes Labs-Storage magazine Quality Award survey for enterprise arrays:
  • EMC Corp. Symmetrix 3000 Series, 8000 Series,DMX Series or DMX-3 Series
  • Hewlett-Packard (HP) Co. StorageWorks XP Series
  • Hitachi Data Systems (HDS) Lightning 99xx V Series or TagmaStore Universal Storage Platform (USP) Series
  • IBM Corp. ESS TotalStorage ("Shark") F Series, 800 Series, 800 Turbo Series or DS8000 Series
  • Sun Microsystems Inc. StorEdge 99xx Series
  • 3PAR Inc. InServ Storage Server S400 or S800*
*Too few responses to be statistically relevant

Because system reliability is so important to IT consumers, Diogenes Labs and Storage magazine have again teamed up to determine which enterprise arrays (see "Products in our survey," at right) best deliver on their promise. These awards are determined by a detailed survey sent to Storage readers that quantifies users' experiences in five categories:

  • Sales-force competence
  • Product features
  • Initial product quality
  • Product reliability
  • Technical support
We find this methodology particularly compelling because it's not a popularity contest with an outcome predetermined by market share. Respondent lists are strictly controlled and individual responses are screened to ensure their validity. Results from the surveys are tabulated and analyzed by an independent statistician who has no connection to the storage industry (see "About the survey" and "Survey respondents"). Vendors aren't allowed to nominate products and they're not notified that the surveys are taking place. The results of the surveys are disclosed to vendors at the time of their publication in Storage.

The winner
Last year, Hitachi Data Systems (HDS) Corp. captured the first Diogenes Labs–Storage magazine Quality Award for enterprise arrays, winning four out of five categories. In the 2006 edition of the survey, HDS has consolidated its claim as the top-rated array by repeating as the winner, sweeping all five categories. Hewlett-Packard (HP) Co. moved up from its third-place finish in 2005 to a solid second, placing second in every category. EMC Corp., which was fifth last year, moved up two notches to third, while IBM Corp. remained in fourth. The biggest surprise was Sun Microsystems Inc., which was a strong second in 2005 but fell to last place this year. We also included 3PAR Inc. in our survey this year, but it didn't garner enough responses to be statistically valid for the award.

Dean Register, an IT integrator at a Midwestern energy company, recently replaced the company's IBM TotalStorage ESS 800 with an HDS TagmaStore Universal Storage Platform (USP) 1100. "Administratively, it [the IBM array] was relatively easy to use," says Register, but "the box did not scale very well; you couldn't load it down." The new TagmaStore is a different story. "The Tagma is a genuine industrial-strength box that was built for speed," he says.

The TagmaStore USP system was too new to be included in our 2005 survey. Although TagmaStore was included in this year's survey, it accounted for only 41% of the HDS-related responses. In contrast, EMC's new DMX systems constituted 59% of its respondents. IBM also had a very low rate of new system responses, with just 29% related to the newest DS systems and the balance for various generations of "Shark" arrays.

Ranking of enterprise arrays

Scores drop
This year's scores are generally down overall from last year's. In 2005, HDS won with an aggregate score of 5.83, but fell to 5.76 this year. IBM declined to 5.05 (from 5.41), while Sun dropped below the 5.0 mark to 4.94 (from 5.52). HP and EMC, however, increased their aggregate scores to 5.54 and 5.27, respectively, from 5.47 and 5.19. Statements are scored on a 1.0-8.0 scale, with 4.5 as the mean.

Along with a number of readers, we were surprised with EMC's low scores on last year's survey given that it's the acknowledged market share leader. We hypothesized that EMC's user base is polarized, with some users truly liking EMC's high-end arrays and others less enthusiastic about them. As a result, we've added another layer of analysis called "top-two box" and "bottom-two box" analysis. Using this technique, we can see how many times a vendor was rated with the top-two and bottom-two responses. Conceptually, this analysis is the opposite of an average. Indeed, our hypothesis played out. EMC had a high rate of top-two scores (51%), second only to HDS (59%). However, these high scores were offset by bottom-two scores of 8% for EMC vs. 5% for HDS. HP had the lowest rate of bottom-two scores with just 4%. Sun had the highest number of bottom-two scores with 11%.

In 2006, we also extended our demographic analysis. We asked respondents to tell us how much capacity each of their systems had. From this information, we learned that more than half of IBM, Sun, HP and even HDS respondents have less than 100TB on those companies' systems. In contrast, more than half of EMC respondents had more than 100TB. Indeed, EMC had 39 respondents (28%) with more than 200TB; HDS had 16 respondents with more than 200TB vs. only single-digit respondents for HP, IBM and Sun. Seventeen percent of respondents had four arrays, but 3% reported having 17 arrays.

Ranking of enterprise arrays

Sales-force competence
In 2005, the only category HDS didn't win was sales-force competence. That changed this year, despite HDS' score in this category actually falling from 5.68 to 5.49 (see "Sales-force competence," at right). HDS' highest score in this category was in response to the statement "My sales rep is easy to negotiate with" (5.71). Sun, the previous category winner, had its score tumble more than a full point from 5.73 to 4.61.

Sales is often a sticking point among users, capable of making an otherwise good vendor relationship turn sour. Brown-Forman Corp., Louisville, KY, engaged a consulting firm to help with the purchase of an EMC Symmetrix DMX-3. "We negotiated not only the price of the DMX, but we negotiated per-terabyte pricing at certain tiers over a four-year period," says Steve Greenleaf, senior systems engineer at Brown-Forman. "We were pretty aggressive in our negotiations."

The biggest turnaround in the sales-force category was HP, which went from fifth place last year to second this year despite only a fractional scoring rise from 5.20 to 5.21. Nevertheless, HP was the only vendor to remain steady as all others declined in their ratings. EMC and IBM (with scores of 5.19 and 5.18, respectively) were grouped very closely to HP. EMC could have surpassed HP in this category, but its score was dragged down by a 4.70 for the statement "My sales rep keeps my interests foremost." Sun experienced rare negative ratings (4.47) to this same question, as well as for "My sales rep is easy to negotiate with" and "My sales rep is flexible." For Sun, this is a surprising slip in just one year. The ratings decline, we hypothesize, is a result of disruptions due to integrating significant acquisitions (e.g., StorageTek) and major corporate reorganizations.

The repercussions of a sale can resound long after the ink has dried on the contract. "It was a sales error that we're suffering right now," says Mark Gangola, manager of data center operations at New Century Mortgage Corp. in Irvine, CA, referring to the replication software his company bought with its EMC Symmetrix DMX1000. "Every time we've used it, it's broken something," says Gangola. They've temporarily pulled the plug on the $175,000 software package.

Ranking of enterprise arrays

Product features
To assess product features, we asked respondents about their experiences with software functionality such as snapshots, mirroring and management features. Once again, this category ran counter to the trend of overall lower scores. HDS won this category with a 5.94, the highest category score in this year's survey (see "Product features," at right). Whereas scores above 6.0 to individual statements in 2005 were good, they were more common than in the 2006 survey. HDS had four 6.0-plus scores in the product features section, including, "This product's snapshot features meet my needs" (6.07), "This product's mirroring meets my needs" (6.04), "This products remote replication meets my needs" (6.16) and "This product scales to meet my needs" (6.10).

We compared this category's scores for HDS, HP and Sun because all of these systems function identically; HP and Sun sell OEM versions of the HDS product. Although the scores were different, they did track together, albeit by an offset amount. That means that if the scores were placed on a graph, they would follow a similar track. Although one might expect them to be identical, it's likely that nonproduct factors (e.g., sales-force competence, technical support, professional services) impact a user's perception of the product's performance. These scores seem to support that hypothesis.

Ranking of enterprise arrays

Initial product quality
The initial product quality section of the survey assesses not only how easy the product was to install and configure, but also the degree to which respondents were satisfied with any professional services provided by the vendor. The services component of enterprise-array product delivery tends to be quite high. In 2005, we used the statement "This product does not require professional services"; however, we felt the statement unfairly prejudiced responses against "high-touch" products. Indeed, many users prefer the high-touch engagement model. For this year's survey, we substituted the statement "I am satisfied with the level of service for this product." EMC's score on this issue rose from 3.22 on last year's survey to 4.80. But users gave the company a low score of 4.12 for "This product requires little vendor intervention." Overall, EMC's score for the category rose from 4.46 to 4.83, but still landed it in last place for the category, just barely trailing Sun (4.84) and IBM (4.86) (see "Initial product quality," at right).

Brown-Forman's EMC Symmetrix DMX-3 worked without a major hitch from day one. "Once the hardware rolled in the door," says the firm's Greenleaf, "it was pretty much ready to use."

While the other three products in this category were rated below 5.0, HDS and HP broke away from the pack with solid 5.80 and 5.58 scores, respectively. No product scored above 6.0 to any statement in this section, but HDS came close with a 5.96 for "This product offers good value for the money." HP rated second on this statement with a 5.15, followed by IBM (4.84), EMC (4.75) and Sun (4.56). Last year, three of HDS' six scores in this category topped 6.0.

Ranking of enterprise arrays

Product reliability
Our product reliability section confirmed that all of the products in this category perform well in the data center. Every product scored above 5.0, led by HDS (5.92) down to IBM (5.04) (see "Product reliability," at right). Nevertheless, the overall ratings in this category fell on a year-over-year basis. In 2005, scores ranged from 6.16 (HDS) to 5.41 (Sun).

One area that captured our attention in nearly all of the product categories in previous awards is patch management. In most surveys, we've found this area to be among the lowest rated and a differentiator among vendors. This year, HP (6.04) bested HDS (5.97) for the statement "This product requires very few unplanned patches." However, HDS received higher marks (5.86) for "Patches can be applied non-disruptively" vs. HP's 5.76. The statement "This product meets my service-level requirements" received generally high scores, with HDS taking the top spot with a 5.97.

Despite his software woes, New Century Mortgage's Gangola is more than happy with his EMC array's reliability. "We've had phenomenal up time," says Gangola. "Overall, the fact that we're at four-nines-plus [means] the equipment has been pretty good."

Ranking of enterprise arrays

Technical support
None of our surveys has revealed significant differentiators among vendors based on technical support. In 2006, HDS, HP and EMC all maintained scores above 5.30 (see "Technical support," at right). IBM and Sun, however, slid to 5.08 and 4.81, respectively. For the statement "Support personnel are knowledgeable," respondents awarded HDS a 5.61 and HP a 5.53. When we posed the statement "Vendor takes responsibility for the problem," scores ranged from a 5.68 for HDS down to a 4.72 for Sun.

"The fact that we don't have to worry about a disk going out or something happening helps us sleep at night," says Brown-Forman's Greenleaf about his experiences with EMC's technical support. But Tony Bergen, manager of server technologies for The North West Company Inc., a Winnipeg, Manitoba-based food and merchandise retailer, had a completely different experience with support for his IBM Shark. He says local IBM support staff "didn't seem to be the sharpest knives in the drawer."

Final analysis
Most IT users form an overall opinion about a product based on their varied experiences. To assess these opinions, we examined two different items. The first item, head-to-head analysis, compares the scores a user gave when evaluating two or more different systems. In 2005, IBM and Sun had the highest win rates, coming out on top in 100% of their head-to-head comparisons. But 2006 wasn't as favorable for these vendors, as they trailed the field with 32% win rates. HDS had the highest win rate (63%). In the head-to-head comparison in 2005, EMC was last, but jumped to second place in 2006 with a 53% win rate. HP placed third with a 33% win rate, just barely above that of IBM and Sun.

We also asked respondents to tell us if, all things considered, they'd buy the same system again. "We're familiar with the EMC technology and we didn't really have any compelling reason to move away from it into HDS or IBM," says Greenleaf. Apparently, other respondents shared this sentiment. In roughly two out of three cases or better, respondents indicated they would make the same decision again. Brand loyalty remains high in this category. HDS again led the group in this important indicator with 88.4% positive responses, followed by EMC (79.3%), HP (69.8%), Sun (68.8%) and IBM (62.2%) (see "Would you buy from this vendor again?," at right). Congratulations to HDS for retaining bragging rights for this important category for another year.

About the survey
The Diogenes Labs-Storage magazine Quality Awards are designed to identify and recognize products that have proven their quality and reliability in actual use. The results are derived from a survey of qualified Storage readers who assessed products in five main categories: sales-force competence, product features, initial product quality, product reliability and technical support. Our methodology incorporates statistically valid polling that eliminates market share as a factor. Our objective is to identify the most reliable product on the market regardless of vendor name, reputation or size. Products are rated on a 1.0-8.0 scale, where 8.0 is the most favorable score.

Survey respondents
In this year's enterprise array Quality Award, we tallied 638 valid system evaluations from 371 respondents. This response rate was 24% higher than last year, and yielded a 5% margin of error with a 95% confidence factor. We noted a statistically better distribution of finalists this year, with the number of responses ranging from 138 for EMC Corp. to 32 for Sun Microsystems Inc. The largest number of responses came from the healthcare industry, followed by IT services and financial services; last year, financial services was the largest group. As with all of our earlier surveys, the majority of respondents had operationally oriented titles, with only 6.8% being senior executives (chief information officer or chief technology officer). Respondents were asked to evaluate only those systems with which they had personal experience within the last year.

This was first published in September 2006

Dig Deeper on Disk arrays



Find more PRO+ content and other member only offers, here.



Forgot Password?

No problem! Submit your e-mail address below. We'll send you an email containing your password.

Your password has been sent to:



  • Flash technologies remain hot in 2016, experts predict

    Experts predict solid-state technology will remain hot in 2016, leading to the demise of high-speed hard disk drives, as ...

  • Tintri VMstore T5000

    Like all of its VM-aware storage systems, Tintri’s first all-flash array -- the Tintri VMstore T5000 -- allows admins to bypass ...

  • SolidFire SF9605

    The high-capacity SolidFire SF9605 uses SolidFire’s Element OS 8 (Oxygen) to deliver new enterprise features such as synchronous ...