Quality Awards IV: It's a tie--EMC and NetApp share enterprise array honors

In the four years we've conducted our Diogenes Labs-Storage magazine Quality Award for enterprise arrays, we've never had co-winners ... until now. EMC Corp. rode to the top on very strong scores in the product features and reliability sections, while co-winner NetApp was a model of consistency.

This article can also be found in the Premium Editorial Download: Storage magazine: Betting on an enterprise-level virtual tape library (VTL):

In our closest Quality Awards finish yet, EMC and NetApp tied for first, with Hitachi Data Systems and Hewlett-Packard only a few fractions of a point behind.

In the four years we've conducted our Diogenes Labs–Storage magazine Quality Award for enterprise arrays we've never had co-winners until EMC Corp. and NetApp tied for first place this year.

EMC has risen steadily in this survey and enters the winner's circle for the first time. Similarly, NetApp rose from prior disappointing results in the enterprise array category to join EMC at the top. Both companies posted overall scores of 6.53 on the survey's 1.0 to 8.0 scoring scale. NetApp is on a roll after winning top honors in the most recent NAS survey for its enterprise and midrange systems. Hitachi Data Systems Corp., the winner of all three previous enterprise array awards, fell to third place with a score of 6.50. And to further emphasize what a horse race this year's competition was, Hewlett-Packard (HP) Co. trailed Hitachi by a mere 0.01 of a point. IBM Corp. finished fifth with a 6.39.

The competition was close across the board (see "Products in the survey," below). Our survey includes five sections and within those sections we had four different companies post the high score (see "About the survey," below). EMC was the only company to win two categories: product features and product reliability. Surprisingly, previously dominant Hitachi Data Systems didn't win a single section. Sun Microsystems Inc. (sixth) and 3PAR Inc. (seventh) were the other two companies that didn't win a category, although Sun had a respectable 6.32 overall finish. 3PAR, in its first year as a finalist, experienced just how brutally competitive this category can be. While its 5.94 final score indicates strong user satisfaction, it was the only company to finish below 6.0, coming in last in all categories.

Average scores were slightly higher than last year, a trend with recent surveys. EMC's and NetApp's winning score of 6.53 was the highest overall tally we've seen in any of our enterprise surveys. In last year's survey, 30% of EMC respondents reported having more than 200TB of data, earning the firm the designation as the group's "heavy lifter." While 30.5% of EMC users topped the 200TB mark this year, Hitachi Data Systems nudged past EMC to claim the "heavy lifter" crown with 30.7% of its respondents having that amount of data. Across the board, large-scale sites increased significantly. All but one of the vendors in the survey had at least 20% of its users in the 200TB-plus range, the exception being 3PAR with 5% (one user).


Products in the survey: The following enterprise array products were included in the survey: 3PAR Inc. InServ Storage Server S400/S800, EMC Corp. Symmetrix 3000 Series/8000 Series/DMX/DMX-3/DMX-4, Hewlett-Packard (HP) Co. StorageWorks XP Series, Hitachi Data Systems Corp. Lightning 99xx Series or USP/USP V/USP VM Series, IBM Corp. ESS ("Shark") F Series/800 Series/800 Turbo Series/DS8000 Series, NetApp FAS3000/FAS6000 Series, Sun Microsystems Inc. StorageTek 99xx series*


About the survey: The Diogenes Labs–Storage magazine Quality Awards are designed to identify and recognize products that have proven their quality and reliability in actual use. The results are derived from a survey of qualified Storage readers who assess products in five main categories: sales-force competence, product features, initial product quality, product reliability and technical support. Our methodology incorporates statistically valid polling that eliminates market share as a factor. Indeed, our objective is to identify the most reliable products on the market regardless of vendor name, reputation or size. Products were rated on a 1.0 to 8.0 scale, where 8.0 is the best score.

A total of 379 respondents provided 592 system evaluations. This response rate yielded a +/- 6% margin of error with a 95% confidence factor. As usual, EMC Corp. had the largest response group with 167 system ratings. NetApp was second with 119 system ratings and 3PAR Inc. had the least with 20 ratings, which was still well over the minimum statistical threshold. Of all respondents, 63.3% had only one vendor, 23% had two vendors, and the remainder had three or more.

Respondents' company size, based on annual revenue, split relatively evenly at the $1 billion mark, with half above and half below that figure. Of those surveyed, 26.4% belong to organizations with $10 billion or more in revenue, and 25% were from organizations with less than $100 million in revenue. IT services had the largest industry sector representation with 20.9% of respondents, followed by financial services with 16.9%.

* All of the above products received enough responses on the survey to be finalists.

Click here for a chart of
how Enterprise Arrays scored (PDF).

Sales-force competence
"They were great," says Jim Bauer, director of network and infrastructure services at Bates College in Lewiston, ME. "The folks from NetApp understand what we're looking for and understand their products really well."

The sales-cycle experience plays an important role and often lays the foundation for the ensuing user experience. Scores for the sales-force competence section of our survey generally correlate closely with overall results, but not this time. HP, which placed fourth overall, ranked highest for its sales force with a 6.43. NetApp wasn't far behind with a 6.39, followed by Hitachi Data Systems (6.38) and EMC (6.35). In this section, we asked users to tell us their experience regarding sales-force knowledge, flexibility and negotiability.

For many users, the sales process often involves a reseller and the product manufacturer. "We go through a reseller, but usually the reseller brings in resources from Hitachi," says Bryan Moore, storage manager for enterprise IT operations at Scripps Networks in Knoxville, TN. But even with a reseller in the picture, the vendor still plays a pivotal role. "Whenever we have anything outside of normal growth," says Moore, "we bring [Hitachi] in and they assess what our needs are and try to create the best possible fit."

James Odak, senior global systems engineer at Chicago's Wm. Wrigley Jr. Co., worked with both his reseller and HP to acquire the company's StorageWorks XP10000 and XP12000 arrays. His reseller "brought in the HP guy to help them out with the sale," he says. "They did a good job." Odak had a good idea of what Wrigley needed, but used his sales resources to confirm his choices. "I kind of tell them what I really need and they just sanity-proof it," he says.

HP's sales-force competence score was the lowest high score of any survey section. Yet no vendor knocked the cover off the ball in any particular area. Most vendors did well on the statement "My vendor's sales support team is knowledgeable," with NetApp receiving the highest score. The statement also provided EMC with its highest category score (6.58). HP received its highest score (6.63) for "My sales rep is flexible."

Product features
With a solid 6.72, EMC had a comfortable lead over the rest of the field in the product features section. Here, we asked users how well a product fit their particular needs in areas such as snapshots, mirroring, remote replication and scalability.

Wrigley's Odak has found his two HP XP systems slightly tougher to use than HP's midrange EVA line. "The EVA is extremely easy to manage--the XP takes a bit of understanding and knowledge before you can figure it well," he notes. Still, keeping the XPs up and running is hardly onerous for Odak. "Unless you're changing or adding disk constantly, there really isn't a whole lot to operation," he says.

Scores above 7.0 were hard to come by in this survey, but EMC came very close in this section with a 6.98 for "This product scales to meet my needs." EMC users apparently value the product's features in total better than its individual attributes, as it scored a 6.86 for "Overall, this product's features meet my needs." Individual responses to its snapshot, mirroring and replication features were solid, but not as high.

NetApp placed second in the section with a 6.57, and received its best section score for "This product's snapshot features meet my needs." Users also like the product's scalability, giving it a 6.73. In the reverse of EMC, NetApp users gave higher marks to individual features than to the overall system.

"It's been great," says Bates College's Bauer, referring to the overall feature set of his dual NetApp FAS3040s, but he's particularly impressed with the boxes' data protection features. "The snapshot takes five seconds at most; to start and stop the database takes three or four minutes," he notes.

One statement in this section, "This product is interoperable with other vendors," received lower scores relative to other areas across the board. EMC had the highest score of the group with a 6.73, while NetApp had its lowest section score (6.19). Only 3PAR fared worse with a 5.94.

Initial product quality
This section assesses vendors regarding installation ease, degree of professional services involvement, ease of use and value. NetApp's 6.47 narrowly edged out HP's 6.45. This was one of two sections in which all vendors scored above 6.0.

Again, how well a vendor works with its local resellers can make a big difference. "There was a NetApp engineer and one of the local engineers," says Bauer, describing his FAS3040 installations. Moore at Scripps Networks had a similar experience with the role his vendors and reseller played for the installation of his Hitachi Data Systems and 3PAR arrays. "Both are very involved in installation, and they don't leave until we're happy," he says.

NetApp apparently has the best out-of-box experience, receiving its highest section scores for "This product was installed without defects" (6.77) and "This product was easy to get up and running" (6.57). NetApp also had the highest score for "This product is easy to use" (6.49).

EMC trailed closely in most areas of this section, registering its highest score for "This product was installed without defects" (6.76). If EMC had an Achilles' heel in this section, it appears to be related to the statement "This product requires very little vendor intervention." Its 5.91 score was a rare sub-6.0 score for EMC in the entire survey. With a 6.0 or higher score on this statement, EMC would have been the outright survey winner. Of course, a higher score on any question would have pushed EMC beyond NetApp to be the outright winner, but this is the most obvious example of where the company was oh-so-close.

The statement "This product delivers good value for the money," is normally a bellwether to the overall order of finish. But this time, the results were somewhat upside down. Hitachi Data Systems was deemed the "best value" by its users with a 6.48 for this statement, and Sun was second with a 6.46. (Sun re-brands Hitachi's systems). HP, another Hitachi OEM, placed third with a 6.34. NetApp was fourth with a 6.30, followed by IBM (6.27), EMC (6.23) and 3PAR (5.88).

Despite its unimpressive ranking in this section, Moore likes 3PAR's value. "It's pretty cut and dry; when you buy it, features that are separate or unavailable in other platforms are simply built in," he notes.


Product reliability
EMC led the product reliability section with its highest section score (6.85). This was also the highest rated section of the survey; Hitachi Data Systems closely followed EMC with a 6.81, while NetApp scored 6.75. 3PAR posted its best score of 6.16.

EMC scored the only 7.0-plus rating in the entire survey for "This product experiences very little downtime" (7.08). Hitachi came very close with a 6.97 for the statement "This product requires very few unplanned patches." NetApp also narrowly missed a 7.0 score for "This product meets my service-level requirements" (6.96).

The statement "Patches can be applied nondisruptively" has been a key indicator in the past and was so again. EMC had the highest score on this statement (6.77), narrowly beating Hitachi's 6.75. NetApp trailed with a 6.60, but that was still good enough to be in the top three.

"The XP is rock solid," says Wrigley's Odak. "We've never had any downtime whatsoever due to the XP."

"It's never been down since it's been powered up," says Moore of one of Scripps Networks' Hitachi Lightning arrays, which was first fired up four years ago. Scripps' 3PAR InServ Storage Server S400 hasn't achieved that kind of longevity yet, but in its nine months or so of operation, Moore reports they "haven't had any problems with it."

The statement "The vendor provides comprehensive upgrade guidance" often generates below-average responses. This was true for most vendors, but EMC and NetApp bucked the trend with ratings of 6.77 and 6.76, respectively.

Technical support
IBM's support organization helped deliver a section win with a top score of 6.55. Technical support scores have never correlated closely to the order of finish and this survey was no exception. IBM had its best rating in this section for "Vendor provides support as contractually specified" (6.89). It also had the highest score in the group for "Problems are resolved in a timely manner" (6.74).

According to our survey, Hitachi has the most knowledgeable support group. It received a 6.89 for the statement "Vendor's support personnel are knowledgeable." EMC was second with a 6.61. EMC's only other slip in the survey was for "Problems rarely require escalation," where its 5.96 was solid but still below the 6.0-plus norm. Nevertheless, it received a 6.30 for "Problems are resolved in a timely manner."

Enhancing tech support with services like automated phone-home services helps to resolve problems and avoid bigger ones. "So far, we haven't had to have any onsite service," says Bates College's Bauer of his two NetApp arrays. Bauer says NetApp's phone-home monitoring service sent replacement disks before the suspect disks actually failed.

Odak had a similar experience with his HP XP systems' phone-home support. "As soon as something happens, they know it right away and they're on it right away," he says.


But will they come back for more?
Based on the results from our question asking if respondents would be willing to purchase the same system again, vendors shouldn't count on stealing very many customers from each other. The "Yes" response rates to this question were high across the board. HP had the highest "Yes" rate with 91% of its customers indicating they would make the same purchase decision again. Hitachi received a rate of 89.2%, followed by Sun (86%), EMC (85.6%), NetApp (81.5%) and IBM (80.9%). 3PAR had the lowest rating at 70%.

Co-winners EMC and NetApp achieved their identical scores through different means. EMC rode to the top on very strong scores in the product features and reliability sections, while NetApp was a model of consistency. NetApp was the only company to finish in the top three of every section, even though it had the high score in only one section. Nearly every vendor demonstrated some positive attributes in this survey, as well a few areas where they would probably like a do-over. That's a good reason to do it again next year.

This was first published in August 2008

Dig deeper on SAN management

Pro+

Features

Enjoy the benefits of Pro+ membership, learn more and join.

0 comments

Oldest 

Forgot Password?

No problem! Submit your e-mail address below. We'll send you an email containing your password.

Your password has been sent to:

-ADS BY GOOGLE

SearchSolidStateStorage

SearchVirtualStorage

SearchCloudStorage

SearchDisasterRecovery

SearchDataBackup

Close