Storage vendors, like other purveyors of high-tech products, tend to tout their wares with lengthy laundry lists of product specs: speeds, feeds and features--the traditional "wow factors" of technology marketing. While raw capability may be a good gauge of a product's potential usefulness, nothing compares to the test of time and real-world user experiences. The Diogenes Labs-Storage magazine Quality Awards endeavor to quantify those factors by giving storage pros a chance to share their hands-on experiences related to several key storage product categories. In the inaugural installment of the Quality Awards process, respondents completed a survey on enterprise-class storage arrays and then several survey respondents were interviewed.
As the only award attempting to quantify the quality of storage products, the Diogenes Labs-Storage magazine Quality Awards differ significantly from other industry awards. Other awards are primarily popularity contests, with users typically voting for the product with which they are most familiar. As such, these awards become little more than proxies for market share, with established market leaders frequently snaring top honors. The Quality Awards seek users who have practical experience with specific products to gauge their levels of satisfaction.
Four main factors contribute to a purchase decision: suitability, price, TCO and reliability. This model fits nearly all products, from mice to mainframes. Of these factors, reliability is the most difficult to quantify. On the Quality Awards survey, service and reliability questions were posed in five critical categories:
- Sales competence
- Product features
- Product quality
- Product reliability
- Technical support
The winner is ...
No need to waste time with envelopes. The winner is Hitachi Data Systems (HDS) Corp.'s Lightning storage arrays. In what could be considered a runaway, HDS' Lightning received the highest ratings from users for product features, quality, reliability and technical support. "The HDS products are the best of breed in service, reliability and set the standard in disk products," notes one survey respondent.
Other storage arrays included in the survey were EMC Corp.'s Symmetrix line, Hewlett-Packard (HP) Co.'s XP arrays, IBM Corp.'s ESS and DS series, and Sun Microsystems Inc.'s StorEdge 9900 series. Given that the second and third place vendors, Sun and HP, respectively, resell versions of Lightning in this category, it's fair to say that HDS "swept" the award (see "Overall ranking," this page).
The sales process initiates the user's product experience, sets expectations, and may determine whether a product is perceived to be successful or not. In many cases, a vendor's sales reps are the primary points of contact for users. Successful sales organizations are consultative in nature, and ensure that their products are appropriately matched to a user's needs. Sales organizations focused on "meeting quota" will be less particular about assessing the application environment. In this section of our survey, we asked respondents to evaluate sales teams' knowledge of customers' businesses and industries, whether they keep their customers' needs foremost, and how easy or difficult it is to negotiate with a vendor's sales team.
Mike Bottger, technology consultant at National City Bank in Cleveland, notes that neither of his array vendors--HDS and IBM--oversold their products. "Whatever we agreed to, whatever we signed on for, each has delivered," says Bottger.
A link between sales competence and product satisfaction was revealed. Sun netted the highest score in the sales competence category and placed second overall, while HDS placed second in the category and was the overall winner (see "Sales force," this page).
Of course, some users simply don't rely on their sales reps for technical prowess. "The sales support team is usually a little bit light, so it's better that we talk to the engineers or even the product managers," notes a senior storage architect at a large communications and Internet service provider. "From a technical standpoint, they're not deep enough," he adds.
In polling respondents about array product features, we focused on software. We asked users to provide their assessment of array management tools, remote replication software, snapshot software and mirroring software. The intent was to evaluate how well particular products met the responding organization's needs, rather than to simply compare feature lists.
Because EMC pioneered much of the functionality found in enterprise-class arrays, conventional wisdom holds that EMC would lead in array software functionality. However, our survey indicates that other vendors have closed any gap that might have existed and have overtaken EMC. While not specifically faulting EMC's feature set, "EMC storage is too proprietary for our continued use," one survey respondent says.
HDS led this category with a score of 5.68, while EMC lagged the field with a 5.40, but this was the narrowest range of any category. Given the scant differences in scores, we conclude that all of the products generally meet user requirements and expectations. Nevertheless, when respondents rated vendors on "This product's management features meet my needs" and "This product is interoperable with other vendor's products," EMC was the only vendor to receive ratings below 5.0 (4.99 and 4.79, respectively). EMC earned a very high 5.97 for "This product's remote replication features meet my needs," while HDS scored a 5.98 for "Overall, the product's features meet my needs." Sun was the only firm to exceed a 6.0 rating for a question in this category, with a 6.14 for "This product's remote replication features meet my needs" (see "Product features," this page).
Storage array quality was measured by asking questions relating to ease of installation, installation without defects, ease of use and the degree to which vendor intervention was required. The category had the widest scoring spread, with HDS (5.88) taking top honors and EMC placing fifth (4.46) (see "Product quality," this page).
However, a closer look at the questions may indicate that the difference isn't as profound as it might appear. For example, when respondents were asked if "This product did not require professional services," EMC ranked at a very low 3.22; however, one must recognize that EMC offers a "high touch" service model that many users find attractive. (All products in this category have a high services component.) Thus, the question might be unfavorable as it relates to EMC's business model. Still, we would expect the results of this question to be offset by EMC's scores for the related questions: "This product was installed without defects" (5.39), and "This product was easy to get up and running" (4.45). Presumably, significant vendor involvement would shield users from most problems, resulting in a higher score. HDS led the pack on both of these questions, with high average scores of 6.28 and 6.14, respectively.
The key question in this category was "This product delivers good value for the money." EMC users responded with a 4.55 rating, while HDS users gave it a 6.07. HP users were also pleased with their systems' value, scoring the question at 5.73.
To assess product reliability, we asked respondents to tell us how well products met their service expectations. We didn't ask about specific uptime numbers because many organizations don't produce metrics with this level of precision on a subsystem basis. Instead, we asked whether the product met service level requirements, their downtime perceptions, frequency of required patches and the level of disruption caused by product fixes.
"The products and feature sets themselves have delivered as promised," notes National City Bank's Bottger of his firm's IBM and HDS arrays.
This was the only category in which a vendor exceeded an overall score of 6.0--HDS received a 6.14 (see "Product reliability," this page). HP, IBM and EMC all scored above 5.50--a very good score. Sun had a respectable 5.41, but the surprise was that Sun didn't track with HDS and HP given the near-identical hardware. Sun's score was pulled down by a 5.0 rating for the question: "This vendor offers comprehensive upgrade guidance." Thus, inadequate upgrade guidance and resulting problems could result in missing some service level requirements that may be unrelated to the product itself.
Of course, all enterprise array vendors claim "world-class technical support" and, based on the survey results, each seems to deliver satisfactorily. HDS again led the group with a 5.76. When asked if "The vendor provides support as contractually specified," the lowest rank was still a very high 5.71 (Sun). We conclude from the data that technical support isn't a differentiating factor for any vendor, and that IT groups will be well supported regardless of which system they choose (see "Technical support," this page).
Interestingly, vendors received the highest scores for "The vendor provides support as contractually specified," which suggests that expectation levels related to support are fairly well defined by vendors and understood by users. However, responses to questions in the rest of the category indicate that users aren't entirely pleased with support beyond what was contractually agreed upon. For example, the overall average scores for support issues requiring escalation and resolving problems in a timely manner were on the low side at 5.16 and 5.32, respectively (see "Technical support ratings (average for all vendors)," this page).
The quality and bureaucracy of tech support is also a point of contention. "We've checked all the easy stuff, so if we're going to open up a call, we have the facility to get to someone who can really help us drive down to the resolution of the problem," says a senior architect for a communications and Internet services provider. Supporting documentation may also be a problem. "The organization of their [EMC's] documentation on the Web site leaves a little bit to be desired," adds this user. "It's all there; it's just a little hard to find." Still, he has only praise for his local support, which he describes as "outstanding."
Seventy-three percent of our respondents worked with only one vendor's products. Twenty-one percent used two different vendors' products, 5% had products from three vendors and 1% had four different vendors' products. "We're going to maintain a dual-vendor presence," says National City Bank's Bottger. "Our philosophy is to leverage a dual-vendor approach to leverage contracts, technologies, etc."
To compare vendors directly, we examined the 27% of cases in which respondents used more than one vendor. This comparison was based on the average rating assigned by the user for each vendor.
We find the "head-to-head" comparisons from these respondents to be particularly insightful because they're truly comparative, rather than an assessment of perception. Of the 25 IBM respondents who use another vendor's product, IBM was preferred in every case. Similarly, Sun was preferred in every such case (nine respondents). HP was preferred in 23 out of 25 cases, HDS in 33 of 41 cases, and EMC in 17 of 52 cases.
Diogenes Labs and Storage congratulate HDS as the winner of the Quality Awards for enterprise-class storage arrays, and thank the hundreds of users who took the time to thoughtfully respond to our survey.
Respondents were asked to rate their level of agreement or disagreement with a series of evaluative statements in each of five categories using a 1.0-8.0 scale. A response of 1.0 indicated strong disagreement, while 8.0 indicated strong agreement. Because the 1.0-8.0 scale doesn't accommodate a neutral response, all responses indicated some degree of either favorable or unfavorable experiences. Responses were required for all questions, although "Don't know" and "Not applicable" responses were permitted. With 4.5 as the mean on this scale, scores below 4.5 are generally negative while those above are generally positive. However, the vast majority of average scores fell between 5.0 and 6.0. Thus, any score near or above 6.0 could be considered exceptional, whereas a score of less than 5.0 would be an area of concern. The survey included products from the top five companies generally recognized as enterprise-class storage vendors. Vendors whose product lines were represented in the survey had no prior knowledge of the survey, and didn't influence the selection of specific products included in the survey.
The users contacted for the survey were drawn from the ranks of Storage subscribers. In total, 300 respondents indicated that they used enterprise-class storage from at least one of the five vendors in our survey. The survey has a margin of error of +/-6% with a 95% confidence factor.
Approximately 79% of the respondents identified themselves as operationally oriented individuals (e.g., storage managers/administrators, systems administrators, programmers/analysts)--those most likely to have practical experience with the systems. Fewer than 5% identified themselves as a CIO or CTO, while nearly 16% indicated "Other." Small- to medium-sized businesses--defined as companies with less than $100 million in revenue--accounted for a surprising 27.1% of the total number of enterprise-array users. Respondents were fairly evenly distributed by company size (revenue), but 58% represented only five industries, the largest of which was the financial services sector.
The Diogenes labs-Storage magazine Quality Awards
The Diogenes Labs-Storage Quality Awards is a joint project of Storage magazine and Diogenes Analytical Labs. It's an ongoing process that surveys users to gather their perceptions related to service, reliability and quality of key storage infrastructure products. The enterprise array award is the first installment of this effort; next month, Storage will report the results for backup and recovery software. Other major product categories will be addressed in the coming months.