Last month and this month, we've been guilty of something I hope you never have to do: evaluating vendors' performance...
claims in the abstract. In our February cover story ("Inside the new Symmetrix") and this month's lead Trends story, ("Symmetrix DMX: Is it hot or not?") we've written a lot about the putative performance advantages of EMC's new Symmetrix DMX architecture, and how it compares to Hitachi Data Systems' Lightning and IBM's Shark.
We did that without ever reading or writing a single bit to any of those devices. Nor did we report any data from anyone who did, outside of the vendors themselves. It's the sad truth, but a critical review of architectures on paper is the best you can do right now.
Although there's a vendor-neutral performance benchmark (the Storage Performance Council's SPC-1), only IBM--of those three vendors--has seen fit, as of this writing, to report results. Perhaps HDS will. EMC hasn't even joined the Storage Performance Council.
Oh, those wacky vendors--when will they learn? We're about to go through many months of claims and counterclaims about performance, largely unsubstantiated by any actual performance data. Are you starting to discount any of these claims? Good.
While I think engineering critiques of architectures can be worthwhile, they are far from definitive. Performance is not theoretical, and there are only two kinds of performance that are worth knowing about. One is an apples-to-apples comparison of products against a standard benchmark, hopefully one that stresses the system in a way that acts as a surrogate for real deployments. The other is a test suite of your choosing running in your environment. If vendors can't supply you with the first and allow you to perform the second, I'd assume they have something to hide.
Speaking of performance, are you tired of flying blind? After all, once you get the darn SAN up and running, iron out all the HBA problems, get your zoning done, etc., what is it you're going to want to do every day? Make sure it's running up to snuff so that you can make good on SLAs. Sounds like performance management to me.
Yet the vendor community seems to pay very little attention to this basic operational need. I have no clue why. Maybe venture capitalists don't get it and won't fund it. Who knows?
Meanwhile, how can a software company sell a tool to improve utilization in the absence of tools to measure performance, and do it with a straight face? If you find out your database storage is at 35% or 65% utilization, how do you know if that's a good or a bad thing in the absence of solid performance data? You don't.You might even want to decrease utilization, spreading the data out over more spindles.
Storage software vendors: It's time to fill out the suite with some real end-to-end performance management tools, starting at the host and ending at the array. What? You can't do that because there aren't standards for devices reporting that? Hmm, maybe you ought to talk to some of hardware guys. Yeah, you know the ones I'm talking about...