This article can also be found in the Premium Editorial Download "Storage magazine: Continuous data protection (CDP) and the future of backup."
Download it now to read this article plus other related content.
Among the key characteristics of SPC and SPEC benchmarks are mandatory peer reviews of all benchmark results, meticulous documentation of the tested configurations, and making test results and test details available for public consumption. To be representative of real-world applications, the workloads they use are derived from common real-world applications, and the SPC and SPEC pride themselves on gauging performance in a way relevant to enterprise computing. Unlike SPEC, SPC publishes the cost of tested configurations with the test results, providing a cost/performance metric. "You really have to look at both performance and cost," explains Greg Schulz, founder and senior analyst at StorageIO Group, Stillwater, MN. "It required a $3.2 million configuration for the IBM SAN Volume Controller 4.2 to get over 270,000 SPC-1 IOPS, or over $12 per SPC-1 IOPS, a relatively unfavorable cost per IO ratio if compared to other SPC-1 benchmark results," he notes.
Without question, industry-standard benchmarks are a great way to get objective, authoritative benchmark results of storage systems from different vendors; but they're not without challenges. Most importantly, their success and effectiveness hinges on storage vendor participation. With more than 60 members, the majority of NAS vendors are part of SPEC. SPC has more than 25 members, including all major storage vendors except EMC. Unfortunately, being a member doesn't necessarily mean participation. For instance, Hitachi
Moreover, storage vendors are very selective regarding what products they benchmark using SPC. Participation in the SPEC SFS benchmark is significantly higher and includes most NAS vendors, in addition to EMC, HDS and NetApp. One of the reasons for the limited participation is cost. "Both SPC and SPEC benchmarks are quite expensive, and [it's] only if we see a clear marketing benefit or value for our end users [that] we participate," says NetApp's Daniel.
This was first published in October 2007