How useful are storage benchmarks?


This article can also be found in the Premium Editorial Download "Storage magazine: Continuous data protection (CDP) and the future of backup."

Download it now to read this article plus other related content.

Among the key characteristics of SPC and SPEC benchmarks are mandatory peer reviews of all benchmark results, meticulous documentation of the tested configurations, and making test results and test details available for public consumption. To be representative of real-world applications, the workloads they use are derived from common real-world applications, and the SPC and SPEC pride themselves on gauging performance in a way relevant to enterprise computing. Unlike SPEC, SPC publishes the cost of tested configurations with the test results, providing a cost/performance metric. "You really have to look at both performance and cost," explains Greg Schulz, founder and senior analyst at StorageIO Group, Stillwater, MN. "It required a $3.2 million configuration for the IBM SAN Volume Controller 4.2 to get over 270,000 SPC-1 IOPS, or over $12 per SPC-1 IOPS, a relatively unfavorable cost per IO ratio if compared to other SPC-1 benchmark results," he notes.

Without question, industry-standard benchmarks are a great way to get objective, authoritative benchmark results of storage systems from different vendors; but they're not without challenges. Most importantly, their success and effectiveness hinges on storage vendor participation. With more than 60 members, the majority of NAS vendors are part of SPEC. SPC has more than 25 members, including all major storage vendors except EMC. Unfortunately, being a member doesn't necessarily mean participation. For instance, Hitachi

Requires Free Membership to View

Data Systems (HDS) Corp. and Network Appliance (NetApp) Inc. are SPC members, but have never published an SPC benchmark. The SPEC SFS is more established than the SPC benchmarks (almost 10 years older), and there's a higher level of expectation for vendors to participate in it than in SPC benchmarks. "We don't see customers ask for SPC or SPEC numbers," says Steve Daniel, NetApp's director of database platform and performance technology. "One of the primary reasons we continue to publish SPEC SFS benchmarks is for historical reasons; we can't just stop without raising questions."

Moreover, storage vendors are very selective regarding what products they benchmark using SPC. Participation in the SPEC SFS benchmark is significantly higher and includes most NAS vendors, in addition to EMC, HDS and NetApp. One of the reasons for the limited participation is cost. "Both SPC and SPEC benchmarks are quite expensive, and [it's] only if we see a clear marketing benefit or value for our end users [that] we participate," says NetApp's Daniel.

This was first published in October 2007

There are Comments. Add yours.

TIP: Want to include a code block in your comment? Use <pre> or <code> tags around the desired text. Ex: <code>insert code</code>

REGISTER or login:

Forgot Password?
By submitting you agree to receive email from TechTarget and its partners. If you reside outside of the United States, you consent to having your personal data transferred to and processed in the United States. Privacy
Sort by: OldestNewest

Forgot Password?

No problem! Submit your e-mail address below. We'll send you an email containing your password.

Your password has been sent to: