Your worst nightmares ... and how to avoid them

It's 10 p.m. Do you know where your data is?

This Content Component encountered an error
This article can also be found in the Premium Editorial Download: Storage magazine: .NET server storage: Friendly or not?:

The American Academy of Family Physicians says nightmares are a common phenomenon affecting both children and adults. Frequent nightmares in adults, however, are considered abnormal, except in the cases of persons who possess thin boundary personalities, take certain drugs or manage enterprise storage for a living. Storage managers commonly manifest stress levels that are at least on par with those of dentists, Middle East peace negotiators...

or sleep-deprived parents of newborn children. An unscientific poll of IT professionals and their confidants in the vendor and reseller/integrator community reveals that their ten most common storage nightmares align fairly closely with the most frequently reported themes of generic nightmares.

Here's my list.

NIGHTMARE 1: chase or attack
Probably the most common nightmare involves being chased or attacked by a shadowy pursuer. Researchers suggest that the pursuer is often a metaphor for a repressed part of the dreamer's personality - a part they are afraid to confront and integrate. Psychobabble aside, storage managers tell real-life tales of how vendors and resellers pursued them relentlessly to sell products, but later refused to help integrate them into their networks.

Mike Linett, CEO of Zerowait, a storage integrator in Newark, DE, says an IT manager once bought a half terabyte of EMC's Symmetrix storage to host SQL and Oracle data. EMC's sales reps were all over the customer, pressuring him to ink the $500,000 deal. "When the hardware arrived, however, the EMC engineers refused to set it up because they said it wouldn't work in his 10/100 [Ethernet] backbone network," he says.

Linett says this problem isn't limited to a specific vendor. Another IT manager for a major outsourcing company had a similar problem involving a Network Appliance acquisition. "NetApp sold them the storage platforms," he says, "but wouldn't certify their deployment with the customer's Inrange switches." Network Appliance reportedly resolved its customer's situation, after some hemming and hawing, "by providing the customer with Brocade [Communications Systems] switches," he says.

As we move toward an era of greater storage commoditization, there are still substantial differences between storage products and their support for network or fabric interconnections. Unfortunately, incompatibilities are the rule - rather than the exception. To avoid this storage nightmare, managers need to work with vendors proactively to identify not only the best storage box for the job - but also the system and networking parameters that determine how readily the platform can be implemented once it's delivered.

NIGHTMARES 2 AND 3: falling into an abyss and car out of control
The next two most common nightmares reported to dream researchers involve free-falling into a bottomless chasm and careening down the road in an out-of-control automobile. Researchers explain the two dreams in similar ways.

Free-fall nightmares suggest that the dreamer feels ungrounded or unsafe in some aspect of their waking life; car-out-of-control dreams suggest that the rapid pace of change in the dreamer's life is causing consternation. There are obvious corollaries to these nightmares in the everyday life of storage managers.

According to Howie Evans, business development manager with integrator Dallas Digital Services in Grapevine, TX, many customers are pursuing storage consolidation projects to forestall just such nightmares.

"At one large dot-com company," Evans says, "the IT manager told us that the storage decision-making process was totally decentralized and had gotten out of hand. Each department was buying different products that would never work together."

While it's usually the CIO's responsibility to navigate around these diversity shoals, at this company, out-of-control buying behavior was running rampant because of frequent CIO turnover, Evans says.

One way to exorcise this nightmare is for the buyer to do their homework. First, get the vendors to reveal their product road maps. Next, test the product in your environment before you buy. If you don't have the in-house expertise, hire a consultant; it's less expensive in the long run rather than buying some gear that just won't work as advertised. NIGHTMARES 4 AND 5: unprepared, late and public nudity
Nightmares with these common themes often underscore the dreamer's performance anxieties or lack of confidence. Storage managers often report crises of confidence with respect to their acquisitions or the performance of their installations. To stop those concerns, users need better management tools, says Ken Barth, CEO of Tek-Tools, Dallas, TX.

Barth says storage resource management (SRM) tools must "cut through proprietary technology to provide useful management information," adding that "customers want to manage storage from a single pane of glass - just one management console. Once you understand the trends in your capacity utilization, and you can drill for specific information, you won't have the anxiety anymore about the decisions you are making."

But how many SRM tools can operate across a diverse SAN environment?

Mark Bregman, executive vice president for product operations with Veritas Software says the absence of effective software tools for managing increasingly complex and heterogeneous storage environments has resulted in storage that's "underutilized and over-mirrored." Until customers "get the industry to cooperate in a common management model," he says, the confidence level of storage managers isn't likely to get much better.

NIGHTMARES 6 AND 7: stuck in slow motion, without a voice and a helpless/ abandoned/crying baby
Shrinks say that these nightmares reflect the dreamer's inability to verbalize true feelings. As storage technology becomes more complex and as products from disparate vendors are required to work together, storage managers are becoming frustrated when attempting to get a vendor to solve their problem or even pick up the phone. On the other side of the fence, vendors are increasingly pointing the finger at other vendors saying, "It's not my problem - it's their problem." No wonder users get the distinct feeling that they're crying for help and no one is listening.

Bill Basinas, software engineering director for Avamar Technologies, Irvine, CA, and a former software engineer for Legato Systems, explains a vendor's perspective. He says, "A few years ago, Fibre Channel SANs created a lot of pain for companies. Networks were breaking down for no reason. Simple tasks like backup and restore were complicated by a complex infrastructure and a lack of knowledge about SANs [and Fibre Channel.] At Legato, we found ourselves spending 80% of the time helping customers sort out problems with their environment, rather than with our software."

According to Basinas, there was a dot-com that "had lost about a half a terabyte of user data and was attempting to recover it with Legato software. Every time they tried, they received corrupted data. Of course, they blamed the backup software."

"What we found," Basinas says, "was that they had created a file system that was too big for the Microsoft operating system. The file system was so big that just doing a file system check on server boot up would cost the company a day of server uptime." As a result, the customer had never checked the file system, "which was entirely corrupt, except for the last 20GB or so." The backup software was doing its job: The bad data being recorded on tape was, in fact, a faithful duplicate of the bad data on the disk."

Luke Fica - an Avamar software engineering manager also formerly from Legato - agreed that the complexity of the storage infrastructure was making it less possible for customers to set blame effectively. And, he adds, along with the increased complexity there will always be stupid user errors (SUEs).

Fica tells the story about "an intelligence agency within the DOD told us that all of their backups were failing and we needed to fix the problem quickly. Because of the nature of the customer's work, we needed to get our guy a top secret security clearance before he could begin his work. Once that was done, soldiers with guns escorted our guy around for two weeks while he tried to find the source of the problem. When he checked the tape autoloader, he discovered that all of the tapes in the subsystem were cleaning tapes [rather than magnetic tapes]. Apparently, a box of cleaning cartridges had been delivered and the customer had slapped a top secret label on each one [obscuring the manufacturer's label that clearly indicated that the tapes were not for recording]. It cost them two weeks of downtime to discover a simple user error," Fica says. NIGHTMARE 8: personal injury or dismemberment
Shrinks blame this nightmare on the subconscious realization by the dreamer that they are neglecting an important part of their life. Storage managers often report another interpretation: Data is disappearing for no apparent reason.

An IT manager for a major Wall Street financial firm reported recently that using a volume management tool to scale a SAN-based virtual volume cost him 640GB of data and nearly his job. "We were using a virtualization software product from a well-known vendor to aggregate physical disk drives into scalable virtual volumes. The plan was to add more physical disk to the drive whenever our application needed more elbow room. But, when we applied the volume re-sizer utility to an existing volume, it obliterated the virtual disk and deleted all the handles and pointers to data on the volume. Fortunately, we were able to recover most of the information from tape."

This frequently reported problem points to an oft-cited foible of many volume virtualization products. Operating systems don't understand volumes can scale, so virtualization software typically operates outside the rules of the OS. In some cases, when the virtualization software's illegal kernel calls are detected by the server operating system, the operating system responds unpredictably.

Veritas' Bregman, whose Volume Manager software is sometimes blamed for such mishaps, reported that it's the vendor's responsibility to identify what operations or procedures place data in potential jeopardy. He noted that Veritas has resolved many potential snafus in its latest release of Volume Manager, which is now in beta. He recommends that consumers heed the hacker's dictum: Read the fine manual (RTFM) before performing any software operation that has the potential for corrupting data.

NIGHTMARES 9 AND 10: being trapped or locked in and drowning
Dream researchers attribute these nightmares to denying emotions and fearing change. Most storage managers can sympathize.

Derek Gamradt, CTO with integrator/reseller StorNet Inc., Plano, TX, reported a number of instances where customers failed to heed the advice of those in-the-know and deployed products that proved inadequate to the task or created unwanted dependencies on vendors.

"One E-book company," he says, "wanted to implement a SAN for the conversion of books to digital formats. They selected Tivoli's SANergy - despite our recommendation to the contrary. Soon after, they found that backup performance was about 300Kb/s against 2TB of data. They tried everything the vendor recommended to fix the problem, but eventually ended up pulling out the software."

"In another instance," he says, "a title insurance company placed 3TB of RAID storage behind its Microsoft NT servers. They haven't been able to do a full backup since, but they are locked into the RAID vendor's products for the foreseeable future."

Such nightmares, according to Gamradt and others will continue to plague storage managers until one of two things occur: either consumers demand better from their vendors and integrators, or vendors agree on common standards that will most likely lead to the commoditization of their product offerings.

The first option is a dream that has yet to materialize; the latter is the storage industry's worst nightmare.

This was first published in August 2002

Dig deeper on Data management tools

Pro+

Features

Enjoy the benefits of Pro+ membership, learn more and join.

0 comments

Oldest 

Forgot Password?

No problem! Submit your e-mail address below. We'll send you an email containing your password.

Your password has been sent to:

-ADS BY GOOGLE

SearchSolidStateStorage

SearchVirtualStorage

SearchCloudStorage

SearchDisasterRecovery

SearchDataBackup

Close