Published: 12 Jun 2007
Consistency is the key to a disciplined storage operation.
A common theme in most IT shops these days is how stressed out the storage department is and how the number of available resources falls short of customer demand. In defense of our brethren, storage environments are getting more complex, data keeps growing at an uprecedented rate and everyone still has only two hands. You can continue to fight fires by adding more bodies to your staff, or you can try to douse the flames once and for all by rethinking the way storage is managed.
Micromanagement is a human quality. The more out of control the issue, the more attention it requires. The greater the number of issues you have to deal with, the more likely you'll be interrupted, which means that something will inevitably slip through the cracks. But all is not lost--you can still go from a reactive stance to proactive mode by making a few subtle changes to your environment.
Consistency. The key to easing storage management is consistency. Whatever you do, be consistent at it. I'm not suggesting that if you're doing something counterproductive you continue to do it, but be consistent in the sense that you treat storage management as a science, not an art. Consistency essentially allows you to create repeatable processes that, in effect, remove some of the intelligence from the process and provide ample opportunity for automation.
Standardization. Standardization goes hand-in-hand with consistency. Standardization complements consistency by making processes repeatable, minimizing stress involving one-offs, and ensuring that all initiatives and management issues are treated the same way. If there's one quality that can be the bane of a storage environment, it's one-off solutions. You may be able to manage your environment consistently without implementing standardization, but you're probably still dealing with one-off solutions, albeit in a somewhat consistent manner. That's hardly the stuff of an efficient operation.
Consistency is never complete without standardization; together, the two can ensure that an operation remains healthy. Just as you monitor your own health, you need to keep your finger on the pulse of your storage operation. If you see the number of one-off fixes increasing, it's time to take action.
Focus your efforts
There are several areas where you can strive to achieve consistency and standardization. How much you can achieve depends on the size of your operation and the resources you have available.
Storage implementation. Right from the drawing board, a consistent storage environment implies that all objects in the environment look identical. The "low-hanging fruits" of standardization include standard LUN sizes, predefined storage tiers for types of applications, a minimal set of vendor and product types, and predefined definitions for fabric ports.
Let's take the case of standard LUN sizes. On the face of it, standard LUN sizes may appear to be a big waste of capacity. But coupled with host-based volume management and proper utilization monitoring, standard LUN sizes can go a long way in maintaining a well-balanced storage subsystem.
Balancing a subsystem isn't a trivial task, but it's easier to balance it from the ground up than to try and convert an existing system that's bent out of shape.
Predefined storage tiers. Storage tiers are meant to ensure that each type of deployed disk has a certain performance (and hence usage) profile that should be matched to the needs of the application. By assigning a list of profiles to storage tiers, you minimize the risk of rogue applications misusing a storage tier which, in turn, can lead to one-offs to remediate performance issues.
Standard register of vendors and product types. Every vendor takes pride in its product portfolio and will go out of its way to demonstrate just how well its product will fit in your environment. But a storage environment is no different than a child's closet that becomes a disorganized mess when more toys are thrown in it. You have to decide upfront how much deviation you can tolerate from a standard register of vendors that meet your requirements. You may be better off working with a product from a preferred vendor whose products may not have all of a competitor's bells and whistles if the tradeoff is a closely integrated and highly interoperable environment. You must also keep in mind that any one-off deviation from your vendor list is likely to result in higher maintenance costs.
That's not to say that you should be averse to introducing new technology into your environment. But the process of introducing new technology should be streamlined, standardized, and done in a product-, vendor- and resource-agnostic manner. This allows all new products to be treated in a balanced way and, more importantly, ensures that when a new technology is introduced into the main production environment, it will blend in efficiently without causing any operational disruptions.
Fabric definitions. There's nothing as tempting as finding an empty port in a switch to plug in a Fibre Channel cable. It doesn't take long to go from a healthy fabric to one that looks like spaghetti. Standard definitions or tiers for different initiator and target types will ensure that the fabric won't become an uncontrollable jumble.
But technology is only half the story. Policies and procedures are the other half of the operational management process and they can't be ignored. Some areas where you should ensure that standards are followed include:
Provisioning. A frequent complaint from storage administrators is that when customers want storage, yesterday is considered too late. But the more hurriedly you try to provision, the more likely a mistake will be made. Responding in "hurry up" mode will also make it more likely that storage gets provisioned inconsistently. A provisioning policy with different levels of severity classified by the required purpose can avoid missteps. Resist the temptation to meet needs on an ASAP basis, but avoid the other extreme of binding the process with red tape.
Publish your operational practices. It's important to make the added effort to publish your guidelines. If more people are aware of your guidelines, you'll have a better chance of ensuring conformity and minimizing deviations. But be realistic. If your existing processes have a turnaround time that doesn't meet the norms of your IT department, it may be time for a reality check. However, it's imperative that all members of the storage team, as well as customers, subscribe to your discipline. You may have to sell them on the benefits to make them advocates of your methodology.
Know your environment. You'll never truly know what shape your environment is in until you gather operational details on a routine basis. Taken one at a time, stop-gap measures may not be so obviously detrimental; collectively they can add up to an environment that's horribly out of shape. By monitoring your environment on a regular basis, you'll be able to detect those one-off fixes early enough to avoid their cumulative effects.
Change control. Perhaps the single greatest threat to effectively managing a storage operation is allowing changes to the environment to occur without proper change control mechanisms in place. A small change in one area can have profound effects on another operation, and seemingly inconsequential changes can accumulate with devastating--and unexpected--effects. If documented, most changes can be rolled back, if necessary.
At the end of the day, you probably won't get a medal for creating and running a streamlined operation. But managing a more consistent and standardized environment will give your blood pressure a break.