 |
Randy Kerns
Partner, Evaluator Group
|
 Randy Kerns is a partner at the Evaluator Group and is responsible for storage area network (SAN) and network-attached storage (NAS) analysis and education as well as company and product strategies. He has over twenty-eight years storage product development, including work for IBM, Fujitsu, Vice President of Engineering at the Array Technology subsidiary of Tandem Computers, and Director of Engineering for Enterprise Disk at Storage Technology Corporation.
|
|
|
Recently there were several announcements of performance benchmark results by disk storage vendors using the Storage Performance Council SPC-1 benchmark. These benchmarks are listed with analysis on the Evaluator Group Web site. Since the benchmark became available to member companies in May, there have been several of the popular storage systems that have had performance benchmarks run. The SPC-1 benchmark covers products that are capable of being attached to open systems (not mainframe) with a synthetic performance test that was constructed by analyzing traces from many different environments. Databases, e-mail applications and online transaction process applications were included in the SPC-1 performance profile.
From a customer standpoint, performance data gives another element in the information needed to make a storage decision. The SPC-1 data includes a performance profile that relates response time to I/Os per second (IOPS) along with a price calculation per IOP based on the supplied cost data. There is a great degree of variability in the cost data since one vendor may use list prices while another may include a discount with the statement that "this is the price at which we would sell the system." The information is invaluable if interpreted correctly and in the context of the overall requirements that a customer has.
Included here are the published results as of December 9, 2002 and a chart showing the results for all products. Obviously not all products available for sale have been benchmarked and some of the products are not even the current generation being offered by the vendor. There are several benchmarks still in process, which will be posted once the results are independently audited. The process that ensures the integrity of the information is explained on the Storage Performance Council Web site. An executive summary and a full disclosure report of the performance results are also available on the Web site along with the cost details.
Chart 1 -- Posted results
Chart 2 -- Response time curve
The response time curve needs to be used carefully -- how far the speedometer goes up to on the car doesn't mean that you will be operating in that range. We recommend as a general rule-of-thumb that a 10ms response time is the threshold to use when examining performance and the customer should also look at the I/O requirement for a given capacity now and for the next three years. Anything outside that range should be ignored as being too slow or you may be paying for something you're not going to use. Obviously, some of these products can scale in performance and all scale in capacity to some point so there is some degree of variability.
Not all vendors that have products have committed to publishing audited benchmark performance data. That's unfortunate for the customers and only when the customers demand it will we see other vendors add their results.
The current SPC benchmarks are a very positive first step. There will be additional benchmarks to cover different application environments and certainly updates to the performance data as new models and new products are released.
