Published: 11 Aug 2004
Steve Duplessie once called Richard Scannell, GlassHouse Technologies founder, "the smartest man in storage." Now, Richard is smart, but I don't think that's what Steve meant. Rather, he was trying to explain why people want to hear what he has to say. You see, Richard has experience, so he can spot best practices. And anyone who bases their decisions on best practices is going to be the smartest person around.
So what's a best practice anyway? And how can you become the smartest, too? Let's start at the top.
More than just a good idea
A best practice is a slippery thing. I've heard people joke that they can make a practice into a best practice just by adding one word. But of course, not every good idea is a best practice. And one person's optimum solution is another's optimized mess! A best practice for one environment might not work for another.
Despite this, it's clear to me that there are special ideas that can be called best practices. I've found that the following three questions can distinguish a true best practice from a mere good idea:
Is it prudent? Let's borrow the Prudent Man Rule from finance: Is this something that a prudent person of discretion and intelligence would do? This simple question eliminates all sorts of oddball solutions.
Is it widespread? It's hard to argue that something that has never been done before could be called a best practice. Cast aside untested ideas, no matter how good they look. Wait for someone else to risk their data and their jobs.
Is it low risk? Ask yourself what the chances--and consequences--of failure are. It doesn't matter if a solution works perfectly most of the time because only the failures will be highlighted.
These are the three questions that can inform many of our daily decisions. Although it wouldn't be much fun to live life this way, this philosophy can help with the big questions. And none of us are being paid to tempt fate.
The four fundamentals
Now, let's talk about some actual best practices for the storage industry. I'll start with four fundamental best practices from the wider IT world, and then zoom in on how they inform the type of storage decisions Ifind myself making.
Best practice 1: Minimize complexity. IT is a tough world. We're ignored as long as everything is working well, and we're in trouble when things start going wrong. In most cases, we have accountability, but not control. The best we can do is to minimize the complexity of the solutions we must support. At least then we have a fighting chance.
This is where the current move toward consolidation and standardization comes from. I've mentioned before how a single administrator can manage a far larger environment if it's standardized. This is also an outcome of tiered storage models--we can stick with a single technology for each of our few tiers of offerings.
Unfortunately, minimizing complexity is the antithesis of a true hacker's heart. Who wouldn't want to play with the latest gadgets? Why not build an elegant interconnected system? But these solutions fail our tests--they're risky, unusual and just plain imprudent.
Best practice 2: Use the right tool for the job. As any woodworker or mechanic can tell you, it pays to use the right tool for any job. Even if it works, it's just a bad idea to force a network-attached storage (NAS) filer to look like a block device, or a storage area network (SAN) to look like a collection of shared files. That's why I never believed in the direct-access file systems (DAFS) protocol and rarely recommend SAN file systems. But sometimes, they're OK.
You see, this best practice is trumped by the first. It's better to minimize complexity and stick to what you know than to press for the optimum solution every time. It wouldn't be right to introduce a SAN to an all-NAS environment for a single database. In that case, DAFS, or another NAS protocol, would be the right solution. But for most people, this is still akin to driving in a screw with a hammer.
Best practices: Consolidate file servers into NAS boxes. Don't use oddball square-peg technologies.
Best practice 3: Prepare for failure. The hacker in me wants to build everything so it will work, but the realist in me knows that I have to make it twice as good. You can't prepare for every eventuality, but you can look for the probable points of failure and harden them.
Many of the fundamental best practices in storage come down to this. The redundancy of RAID 1, the widespread adoption of multipathing software and the use of redundant SAN fabrics are all examples of preparing for the worst. We wouldn't have adopted these practices if we didn't know that failures happen with alarming regularity and that storage is extremely sensitive to failure.
Best practices: Build redundant everything: SAN fabrics, data centers, network links. Rotate tapes off site daily. Have spare parts handy. Don't listen when someone claims that maintenance won't affect production.
Best practice 4: Align expectations with reality. User A has 100GB of data to be restored. So you tell him you can restore it in 10 hours. But what if users B through G come asking for their data at the same time? It's time to adjust everyone's expectations.
This is where concepts like SLAs and business impact analyses come from. Although we don't have real control over user demands, we can at least control their expectations by demonstrating what we can and can't do. And it's up to us to do this in words that they understand, rather than through technical jargon.
Best practices: Write out your SLAs. Get a reporting tool to help you understand the reality of your infrastructure. Don't let your users labor under their misconceptions of your capabilities.
Are you experienced?
Why do people hire consultants? Why do they come to conferences and read trade magazines? Are we (consultants, pundits, writers) really that special? A new consultant posed this question recently, and my experience at Storage Decisions last April provided the answer.
You see, the crowd wasn't really there to see us; rather, they came to learn from the experiences of others. We presenters, writers and consultants are merely a mechanism to collect and spread those experiences. We talk to lots of folks so we can spot a prudent, widespread and low-risk best practice a mile away.
I'm not sure about you, but I don't live a best practices life. There are lots of risky, imprudent, exclusive activities I engage in. But when it comes to work and customers, best practices continue to guide my recommendations.
This September, I'll be speaking at Storage Decisions in Chicago. Everyone I know leaves this conference with a headful of new ideas. Stop by and share yours with me!