|
Among the key characteristics of SPC and SPEC benchmarks are mandatory peer reviews of all benchmark results, meticulous documentation of the tested configurations, and making test results and test details available for public consumption. To be representative of real-world applications, the workloads they use are derived from common real-world applications, and the SPC and SPEC pride themselves on gauging performance in a way relevant to enterprise computing. Unlike SPEC, SPC publishes the cost of tested configurations with the test results, providing a cost/performance metric. "You really have to look at both performance and cost," explains Greg Schulz, founder and senior analyst at StorageIO Group, Stillwater, MN. "It required a $3.2 million configuration for the IBM SAN Volume Controller 4.2 to get over 270,000 SPC-1 IOPS, or over $12 per SPC-1 IOPS, a relatively unfavorable cost per IO ratio if compared to other SPC-1 benchmark results," he notes.
Without question, industry-standard benchmarks are a great way to get objective, authoritative benchmark results of storage systems from different vendors; but they're not without challenges. Most importantly, their success and effectiveness hinges on storage vendor participation. With more than 60 members, th...
To continue reading for free, register below or login
To read more you must become a member of SearchStorage.com
e majority of NAS vendors are part of SPEC. SPC has more than 25 members, including all major storage vendors except EMC. Unfortunately, being a member doesn't necessarily mean participation. For instance, Hitachi Data Systems (HDS) Corp. and Network Appliance (NetApp) Inc. are SPC members, but have never published an SPC benchmark. The SPEC SFS is more established than the SPC benchmarks (almost 10 years older), and there's a higher level of expectation for vendors to participate in it than in SPC benchmarks. "We don't see customers ask for SPC or SPEC numbers," says Steve Daniel, NetApp's director of database platform and performance technology. "One of the primary reasons we continue to publish SPEC SFS benchmarks is for historical reasons; we can't just stop without raising questions."
Moreover, storage vendors are very selective regarding what products they benchmark using SPC. Participation in the SPEC SFS benchmark is significantly higher and includes most NAS vendors, in addition to EMC, HDS and NetApp. One of the reasons for the limited participation is cost. "Both SPC and SPEC benchmarks are quite expensive, and [it's] only if we see a clear marketing benefit or value for our end users [that] we participate," says NetApp's Daniel.
|
 |
|
