Benchmarking the benchmarks
By Alan Earls

Benchmarks, like EPA mileage ratings, offer an appealing if, at times, flawed means of assessing the relative merit of competing products. Unlike EPA ratings, however, there are numerous different benchmarking standards and tests.

According to Arun Taneja, analyst with Enterprise Storage Group, Milford, Mass., individual switch companies generally use their own benchmarks to measure latency and other characteristics of the switch. The same is true of host bus adapters (HBAs).

However, Taneja says the industry benchmarks that dominate SANs are IOmeter and NetBench while for NAS the dominant benchmark has been SPECsfs from SPEC.org. "EMC and NetApp have fought battles on this for several years --unfortunately, SPEC found a serious bug in the benchmark recently and in mid-June put up a note on their web site that SPECsfs should not be used for NAS comparison until the bug is fixed," says Taneja. This doesn't necessarily mean that SPECsfs numbers are suspect because, Taneja suggests, it may simply have been a problem with accommodating new architectures like BlueArc.

"I think SPEC has the opportunity to take the lead in SANs just like they did in measuring system performance with SPECint and SPECfp," says Taneja. "They already have the organizational and technical infrastructure set up and have relationship with relevant vendor community," he adds.

Additionally, for an alternative view of benchmarks

Requires Free Membership to View

-- and some useful downloads -- visit http://www.raid-storage.com/benchmarks.html, a site provided by Advanced Computer & Network Corporation as an adjunct to their main focus (which is the design, manufacture and sale of RAID systems).

Additional Resources:
*Learn more about IOmeter in this searchStorage Administrator tip, "Testing I/O performance".

* We've had some interesting exchange of information on storage benchmarking in our Storage Management Tips and Tricks Discussion Forum. Check it out at http://searchstorage.discussions.techtarget.com/WebX?50@@.ee83ce3.

* And don't forget to send us your own tips! Submit your tip.

About the author: Alan Earls is a freelance writer in Franklin, MA.

This was first published in July 2001

There are Comments. Add yours.

 
TIP: Want to include a code block in your comment? Use <pre> or <code> tags around the desired text. Ex: <code>insert code</code>

REGISTER or login:

Forgot Password?
By submitting you agree to receive email from TechTarget and its partners. If you reside outside of the United States, you consent to having your personal data transferred to and processed in the United States. Privacy
Sort by: OldestNewest

Forgot Password?

No problem! Submit your e-mail address below. We'll send you an email containing your password.

Your password has been sent to:

Disclaimer: Our Tips Exchange is a forum for you to share technical advice and expertise with your peers and to learn from other enterprise IT professionals. TechTarget provides the infrastructure to facilitate this sharing of information. However, we cannot guarantee the accuracy or validity of the material submitted. You agree that your use of the Ask The Expert services and your reliance on any questions, answers, information or other materials received through this Web site is at your own risk.