The Storage Performance Council has released the first set of benchmark results for storage subsystems, but there is a catch. The full results will not be available until next week and some, including EMC Corp., do not believe the results are relevant to real world customer environments.
The SPC claims the new benchmark specification is the first clearly applicable, vendor-neutral process to accurately compare and configure direct-attached or network storage technology. The overall goal of the SPC Benchmark 1 (SPC-1) is to give end-users and integrators benchmark results for every enterprise storage system in the market, but questions have been raised by analysts and vendors as to the real value of the SPC's numbers.
An argument can be made that the SPC test configurations are not relevant to every user environment considering the lack of switches in the test configurations as well as the omission of the impact software has on I/O results in the real world.
The SPC said the non-participation of switching of switching companies hasn't slowed the workload and that it is "in talks" with Brocade Communications Systems Inc.
The SPC maintained that this round of benchmarks is the first of many and that future test configurations may include products from the switch vendors.
"That is the nature of the benchmark. It may not be representative of all environments," said Walter Raizner, general manager of storage products for IBM.
Raizner added that there is always an opportunity for other companies to broaden the spectrum of testing configurations.
EMC Corp., which was an original member of the SPC, left the organization two years ago because it said the direction of the SPC was not relevant to customers.
Ken Steinhardt, director of technology analysis for EMC claimed the benchmark changed between the time EMC left the SPC and the time they made the SPC-1 announcement. "It looked a lot more like our suggestions."
"We're still strongly of the opinion that the benchmark won't address real customer workloads," he said.
Variables like service and support issues and software interaction are omitted from the tests. EMC contends that the lack of such factors skew the test results.
One SPC member took issue with EMC's criticisms. "If you didn't like the benchmark why didn't you stay in the organization and help make it representative rather than taking your ball and going home?" said Flavio Santoni, vice president of sales and marketing, LSI Logic Storage Systems.
EMC said it is not anti-SPC. Steinhardt said the company may explore the organization further in the future.
The full SPC-1 results, including price/performance comparisons, configurations, capacities and speeds will not be fully disclosed until next week, but based on the IOPs results alone, LSI Logic's storage system has rated tops in performance - even besting IBM's Enterprise Storage Server and Hitachi's Lightning.
EMC's Steinhardt said when a tiny LSI Logic box seems to be double the performance of systems from IBM and HDS it says one of two things - either HDS or IBM have been overstating the performance of their products, or the test may be flawed.
End users must take the results with a grain of salt until the tests can be put in context by disclosure of results next week. While one box may appear faster on its face, it may also be twice as expensive as another system tested under the SPC-1.
Christian Ober, principal worldwide storage analyst at Ideas International Ltd., Sydney, Australia, said the SPC-1 benchmark provides a useful and consistent baseline for comparing a broad line of storage technologies just like TCP did from a server environment.
Ober and his colleagues at Ideas International believe the SPC-1 will allow smaller fish in the storage technology pond, like LSI Logic, a chance to position their technology against industry heavyweights like Compaq, HP, IBM and Sun.
According to the SPC's mission statement, it is a non-profit corporation founded to define, standardize and promote storage system benchmarks and to disseminate objective, verifiable performance data to the computer industry and its customers.
SPC members include Adaptec, Compaq, Dell, Evaluator Group, Hitachi, Hewlett-Packard, IBM, Ideas International, LSI Logic Storage Systems, NEC, Sun Microsystems, Unisys and Veritas Software. Noticeably absent from the list are the switching companies like Brocade Communications Systems, Inrange Corp., and McData Corp.
The SPC is offering a set of benchmarking tools to the vendor community, including a multi-platform test kit for the SPC-1 benchmark and what it calls the Goliath utility for stress testing and performance analysis of network storage systems.
Let us know what you think about the story, e-mail Kevin Komiega, assistant news editor
FOR MORE INFORMATION: