Published: 12 May 2006
In our Quality Awards survey, some familiar--and not so familiar--NAS names won in the enterprise and midrange categories.
After seeing the results of the Diogenes Labs–Storage magazine Quality Award for NAS, Hitachi Data Systems (HDS) Corp. users can be forgiven if they indulge in an "I told you so" moment. HDS not only won the enterprise NAS category, but captured the top spot for its enterprise SAN array in a previous Quality Award survey. And it accomplished this twin feat by outpolling much more established competition in each survey.
Network Appliance (NetApp) Inc. is generally the first company that comes to mind when people think of NAS, and the company has parlayed that popularity into solid results in our survey. NetApp was the clear winner in the midrange NAS category and placed a close second to HDS in the enterprise category. With three NetApp boxes installed and another planned as a SAN array replacement, Brian Mitchell particularly likes NetApp's management capabilities. "I'm really pleased with it," says Mitchell, a systems administrator at Atlanta-based TRX Inc., a provider of transaction processing and data integration services to the travel industry. "Basically, it allows me to manage the data, not the devices."
Our results strongly suggest that, as rated by Storage readers, NetApp has the most compelling total NAS product line from top to bottom. This is especially true for standalone NAS because our survey included gateways and NAS blades. In fact, the HDS system in the survey was a NAS blade for Lightning and TagmaStore systems. Hitachi's high-availability features sold Bruce Georgeson, an IT consultant at CGI-AMS, a Phoenix-based consulting firm. "If you configure it right and follow their recommendations," he says, "it's one of the few NAS appliances that have true five-nines."
Large field of products
The voting process for this survey differed from our prior surveys because the responses were clustered around a few major vendors. Of the 21 product lines from 15 vendors included in our survey, only eight products from five vendors generated enough responses to be statistically relevant and considered for the award (see "Products included in the survey"). Nevertheless, these vendors represent the lion's share of the market and, as such, represent a valid sample of the population. Overall, 387 respondents provided 523 system evaluations. One surprise was the number of responses for "Other": 137 respondents indicated this choice for vendor/system despite what we considered an exhaustive list of NAS products. It's likely that the "Other" responses represent a Windows-based NAS system from smaller vendors, combined SAN/NAS products, older models from the listed vendors or non-NAS systems. In any case, the results of these responses didn't figure into the award calculation. The margin of error for the survey was +/- 6% with a 95% confidence factor.
|Enterprise NAS: Would you buy this product again?|
In selecting NAS systems for our survey, we attempted to be as inclusive as possible; the list of 21 products was the longest yet for any Quality Awards survey. The main criterion for inclusion was that the device support CIFS and NFS file systems simultaneously. We included NAS-only systems, as well as SAN-NAS gateways. To differentiate midrange systems from enterprise systems, we placed all Windows-based systems in the midrange category, as well as those scaling to less than 50TB. Systems that can scale to more than 50TB were categorized as enterprise NAS. However, we did consider vendors' product positioning, as this also affects how users regard and evaluate products.
As with our previous awards, we asked respondents to rate their product experiences in five categories: sales competence, product features, initial quality, product reliability and technical support. We also asked if the system was purchased directly from the vendor or from a value-added reseller. Because all of the finalists in both categories are the manufacturers of their respective systems, original equipment manufacturer issues weren't considered.
Within each of the five categories, respondents were asked to respond to a number of positive statements on a scale of 1.0 to 8.0 (see "About the survey"), where 1.0 indicated strong disagreement and 8.0 indicated strong agreement. A rating of 6.0 was the approximate median for both the enterprise and midrange groups in this survey. The preponderance of ratings fell between 5.5 and 6.5. Rating elements above 6.5 can be regarded as exceptional strengths, while those below 5.5 can be regarded as weaknesses.
Sales competence measures the vendor's sales effectiveness in meeting customer needs. The efforts of a sales team not only sets expectations, but can become the foundation of the customer-vendor relationship. When Hawaiian Airlines in Honolulu was shopping for a NAS system, Kevin Kekoa, manager of operations and systems engineering, felt NetApp wasn't very persistent in pursuit of his business, which left the door open for EMC Corp. "[EMC] offered a model that was two levels above what NetApp was offering for the same amount," says Kekoa. That kind of aggressive salesmanship paid off, and today the company is solidly an EMC shop.
Among enterprise systems, sales competence had the smallest differences among scores of any category, with HDS at the top (6.30) and Hewlett-Packard (HP) Co. at the low end (5.67). The scores in this category didn't reveal any real strength or weakness, as all vendors except Dell Inc. (5.32) in the midrange category fell within (or very close to) the normal 5.5 to 6.5 scoring range. NetApp's midrange score (6.52) was the highest among all vendors in the sales competence category, but the lowest top score of the five categories.
We asked respondents how well their sales rep understood their industry, how flexible and knowledgeable reps are, and how easy it is to negotiate with their rep. In almost all cases, the highest marks were given for "My sales rep is knowledgeable." With only one exception, the lowest sales competence marks for both categories were for "My sales rep keeps my interests foremost." While users feel their rep is competent, vendors would be well advised to adopt a more cooperative philosophy to get an edge in the sales cycle.
|Midrange NAS: Would you buy this product again?|
Our survey didn't compare products on a feature-by-feature basis, but rather on how well the product met the respondent's needs. This evaluates expectations vs. reality and is therefore related to sales competence. We asked respondents for their assessment regarding the product's manageability, software functionality (e.g., mirroring, replication), scalability and interoperability. HDS had the highest product feature score among enterprise products (6.75), while NetApp led the midrange with a 6.77. We also asked respondents to rate the statement "Overall, this product meets my needs." In this regard, NetApp's enterprise products lead the field with a 6.85.
TRX's Mitchell has both NetApp FAS900 and FAS3000 series systems in his shop, and particularly likes how the common operating system makes it easy to administer the boxes regardless of their roles in his environment. "All that is the [NetApp] Data Ontap model," notes Mitchell. "I don't have to learn this tool over here for this array and that tool for the VTL [virtual tape library]--it's all together."
In the midrange, NetApp's product features were very highly rated by users. In fact, the NetApp product received scores higher than 7.0 for three statements in this category. They were "This product's snapshot features meet my needs" (7.11), "This product's mirroring features meet my needs" (7.03) and "This product's remote replication meets my needs" (7.03). No other product had as many 7.0-plus scores in the features category. HDS' highest score for any statement was for "This product's capacity scales to meet my needs." This score was significantly higher than NetApp's 6.45 for the same statement regarding its enterprise products. The only score significantly outside the norm on the low end was EMC Celerra NS Series' score of 4.73 for the statement "This product is interoperable with other vendor's products."
|Products included in the survey|
The following products were included in the Diogenes Labs-Storage magazine Quality Award NAS survey.
We expected to see fairly high scores in this category. While our hypothesis held true for the leaders, this category had the widest range of scores. For example, NetApp's midrange FAS200 systems scored 7.03 (the only product score to exceed 7.0 in any category), while EMC's Celerra systems received a 4.97, the only score below 5.0 in any category. HDS led the enterprise products with a 6.82. "There may be a few configuration issues to hammer out at the start," says CGI-AMS' Georgeson about his firm's HDS NAS blade implementation, "but the interface is nice and it has all kinds of options like quotas and defining users and groups."
We asked respondents to rate the products with regard to installation defects, ease of configuration and ease of use, among others. NetApp's midrange products received a 7.0 or higher on every item except for the statement, "I am satisfied with the amount of professional services needed for this product." Perceived overall value is an important indicator of overall product satisfaction and NetApp showed its strength there, too. NetApp's midrange product received the highest rating (7.06) in the initial quality category for the statement "This product offers good value for the money." Among enterprise products, Hitachi had the highest score (6.74) vs. NetApp (6.40) for the "value" question. When it came to the statement, "This product is easy to use," NetApp took top honors in both the enterprise and midrange categories (6.89 and 7.00, respectively).
The product reliability section assesses how well a product performs on an ongoing basis and how easy it is to maintain. We asked respondents about their experiences with regard to service levels, patch management and upgrade guidance. HDS led all products with a 6.84, while NetApp's FAS200 placed first in the midrange group and second overall with a 6.68. Hawaiian Airlines' Kekoa says his firm's EMC Celerra has been very reliable. "It's been awesome," he says. "We haven't had one hiccup with the NAS."
In other Quality Award surveys, patch applications and upgrade guidance were the two most problematic areas. This trend continued for NAS systems. In fact, NetApp's enterprise systems had a rare sub-6.0 score (a 5.3) for "Patches can be applied non-disruptively." That was NetApp's lowest rating for any statement in either the enterprise or midrange categories. Hitachi, however, was an exception to the rule. HDS scored a 7.0 for "This product meets my service level requirements" and a 7.05 for "This product experiences very little downtime." It even scored a 6.95 for "Patches can be applied non-disruptively," one of the highest scores we've seen for this statement. For the same statement, EMC's Celerra scored a 4.43; this is below the median of 4.5, which means that the overall user impression of Celerra patch management was negative.
Similar to our other Quality Award surveys, tech support didn't stand out as either a strong positive or a negative. All of the vendor finalists had acceptable or higher scores, with NetApp's FAS200 midrange system leading all products with a 6.62. Among enterprise products, HDS scored a 6.44, and had the most notable score of 7.00 for the statement "Vendor provides support as contractually specified." Even though Dell's PowerVault 7xxN, EMC's Celerra NS Series and the HP ProLiant DL series Storage Server had scores below 5.5 (5.34, 5.26 and 5.26, respectively), no one "bottomed out" on any item.
The fact that support scores didn't diverge significantly is good news for both users and vendors, as it indicates a generally high level of satisfaction and underscores the importance of solid support for ongoing operations. For example, Steve Mays, chief technology officer at X2 Technologies Inc., a consulting firm in San Rafael, CA, has three Dell PowerVault 745N systems in his shop and recommends them to his clients because he finds them so easy to implement. One of the reasons he sticks with Dell is its responsiveness to support issues. "We've called Dell and said a drive died and, literally, later on that day a courier showed up with the drive," says Mays.
"The local CEs [customer engineers] and SEs [system engineers] are great," says CGI-AMS' Georgeson about the help he gets for his HDS NAS implementation. But he also acknowledges that Hitachi sometimes seems a little shorthanded when it comes to NAS support. "They need some more people," he says.
At the end of our survey, we asked respondents to tell us if, all things considered, they would purchase the same product again. This question shouldn't be confused with buying intentions because it's possible (and often the case) for users to buy based on the philosophy of sticking with "the devil you know." Rather, this question asks people how they would act if the slate were wiped clean and they knew what they know today.
HDS led all products with 90.5% of its users saying they'd buy another NAS product from the company. This is the highest "buy again" percentage for any product in all of our Quality Awards to date. NetApp wasn't far behind with an 87.8% response for its midrange FAS200product, and an 81.4% for its enterprise FAS900/FAS3000 products.
NAS goes head-to-head
We also examined head-to-head rankings when respondents owned systems from more than one vendor. Here, NetApp midrange products came out the highest, winning 20 of 34 head-to-head contests. EMC's Celerra NS series also did very well in this regard, scoring higher than other vendor's products in 17 out of 29 cases. NetApp's enterprise products ranked higher in 24 out of 47 cases. The biggest surprise was that HDS won only six of its 14 head-to-head matchups.
|ABOUT THE SURVEY: The Diogenes Labs–Storage magazine Quality Award for NAS is the fifth in our series of survey-based service and reliability awards. The Quality Awards are designed to identify and recognize products that have proven their quality and reliability in actual use. The results are derived from a survey of qualified Storage readers that assesses products in five main categories: sales competence, product features, initial quality, product reliability and technical support. Our methodology incorporates statistically valid polling that eliminates market share as a factor. Indeed, our objective is to identify the most reliable products on the market regardless of vendor name, reputation or size. Products were rated on a scale from 1.0 to 8.0, where 8.0 is the most favorable possible score.|
Clearly, head-to-head comparisons didn't track to the finish order of this survey. The reason, we believe, is that the comparisons involve all of the survey's competitors, including those with too few responses to be considered for the award. For example, products from BlueArc Corp. and Isilon Systems Inc. were rated very highly, but because of an insufficient number of responses neither was a finalist. However, statistical relevance isn't important for the head-to-head comparisons because it's a comparison of two systems by a single respondent who uses both products, and therefore doesn't rely on statistical calculations.