Customer due diligence in DR planning
By Jon William Toigo
To begin, I want to offer my condolences to the families and friends of all those impacted by the horrific events on and after September 11. Even from my remote vantage point -- in Atlanta attending that ghost town called Networld+Interop -- it was clear that everyone shared in the shock, disbelief, and emotional malaise following the attack. It was difficult to think about business, let alone to do much of it. Suddenly, the technical and political minutiae of the data storage industry seemed terribly unimportant.
That is, until a friend, who was also attending the show, joined me in the CNN Center on Wednesday evening for dinner. At first glance, it was clear that he was agitated. Red-faced and shaking when he slid into the booth, he explained his dilemma.
The triangle: The reseller, the vendor and the customer
As a reseller/integrator of high availability storage solutions, he had provided a storage platform to a customer located in the NYC metro area several months ago. In the intervening months, hard times had hit the storage product supplier whose equipment he had used in the customer's system. The vendor had changed sales strategies, as many have, to favor "direct sales" (sales by the vendor's own sales force) over "channel sales" (those made by third-party integrators and resellers). Doing business through direct sales looks better on the corporate earnings
And, in this case, a customer, as well. As a consequence of the vendor's sales strategy shift, the customer was no longer located in my friend's "territory." Instead, the customer was located in the sales territory of someone from the direct sales organization based on the West Coast.
To make a long story short, the customer had contacted his solution provider, my friend's firm, to seek another storage platform that could be used to mirror data at an alternate site. The customer was fearful that the frequent power and telecommunications outages affecting Southern Manhattan and much of the surrounding area would eventually cause his primary site to become untenable. He was playing "beat the clock" to develop a fallback strategy.
The customer complained that he was getting no support from the storage array vendor. He had been told that the vendor could not supply him with new equipment promptly because of constraints in shipping and airline transportation.
My friend, on the other hand, had exactly the equipment required by the customer: it was just sitting on a shelf at his shop. He was prepared to have his people load the equipment into a truck and drive it to the customer's location. However, the vendor stood in his way. The vendor threatened the reseller with the rescission of his reseller license if he took any such action.
When do extraordinary circumstances mean an exception to the rules?
The N+I conference provided my friend an opportunity to speak with an executive of the vendor company involved. After hearing the story, the executive assured him that an accommodation could be made. A few hours later, that assurance had resulted in a quagmire of politics. Several heated phone calls and a faxed letter from a bureaucrat at the vendor's headquarters stated that my friend had done wrong by going directly to the executive rather than "working through channels." Epithets were slung and everyone's blood pressure hit the boiling point.
In the end, the decision of whether to allow my friend to deliver support to the customer became moot. Either the customer secured a solution elsewhere, or he simply failed: no further voice mails or e-mails were received despite the fact that neither my friend, nor the vendor, had come to his rescue in a timely way.
Some hindsight lessons for the customer, vendor and reseller
As you have probably noticed, I have elected not to include any names in this brief account. Mainly, this is because I have not been able to confirm all of the details and do not wish to stir up more problems in an already difficult situation. The bottom line, however, is that for the customer, the vendor and the integrator, this was not their finest hour.
- The customer should have done due diligence well in advance of the September 11 disaster to anticipate and provide for business continuity requirements. While no one could have anticipated the specific scenario that unfolded, solid disaster recovery planning assumes the interruption of infrastructure and business processes from any number of causes and provides a capability for ensuring that the time-to-data requirements of critical applications can be met.
- The vendor in this case was guilty of inflexibility in the face of an exceptional situation. Good management requires the articulation of policies and rules for doing business and their consistent application across the organization. However, there are exceptions to any rule. In this case, the vendor would have better served the customer by creating a new rule. Without sounding flip, what would have been the harm of deciding that resellers can support customers of the direct sales organization whenever a force majeur or Act of God interruption occurs. The exception would have benefited both the customer and the vendor.
- The reseller is guilty of nothing except for, perhaps, a lack of initiative. The support for the former customer could have been delivered swiftly, without attention to the bureaucrats. Vendors are a dime a dozen and I doubt that the current vendor would have made good on threats given the exceptional nature of the event once the crisis subsided and cooler heads prevailed.
Perhaps the lesson that can be learned from this situation is that IT managers need to expect unexceptional performance from vendors even during exceptional situations. This assumption, while somewhat pessimistic, has the advantage of assuring business survival. The simple fact is that many vendors and solution providers will not deliver best-of-breed support during a crisis for any number of reasons. If some do -- and fortunately many, including EMC and Yipes!, have done so in the current situation -- this will only aid a positive outcome of recovery plan implementation. Plan for the worst is a good strategy going forward.
About the author: Jon William Toigo has authored hundreds of articles on storage and technology and is our searchStorage.com resident expert on storage management issues. Toigo is also the author of storage books, including, "The Holy Grail of Data Storage Management".
Reader comments: Response to Customer due diligence in DR planning
By Robert F. Bryar
Feb 7, 2002
I have read with great interest many of the commentaries espoused from the thoughtful and delightful exploits of Mr. Jon Toigo, and for the most part I have agreed with his witticism. However, after reading his searchStorage tip Customer due diligence in DR Planning I felt that I needed to come forward to correct Mr. Toigo on some of his assumptions, and in particular, challenge his views related to vendors and solutions providers.
Mr. Toigo did a great job of outlining the facts as they were presented to him. Since they are referenced from a "friend" they may have been one sided; however we will not contest the validity of the situation. As a result of his friend's input, Mr. Toigo provided his opinion as to the performance of vendors and solutions providers at the time of crisis. To be succinct, Mr. Toigo states, "The simple fact is that many vendors and solution providers will not deliver best-of-breed support during a crisis for any number of reasons." I would like to provide a few morsels of information that may give cause for Mr. Toigo and others who seem to have a pension for the negative vernacular to see the other side of the coin.
Vendors are not obligated, nor is it their responsibility to assume the liability associated with a customer's inability to properly plan and execute a business continuity plan. Customers must rely on qualified professionals engaged in the practice of business continuity for support. Remember the old adage, "Failure to plan on your part does not constitute a crisis on my part." The fact that a vendor provides business recovery products, such as IBM's PPRC, EMC's SRDF, or Network Appliances SNAPMirror, does not necessarily indicate that he or she is in the business continuity profession. Vendors should not be viewed as resources capable of providing "end-to-end" processes associated with a sound recovery plan unless they are actively providing this service as a profession. Service providers who offer business continuity or replication services would not have left this customer out in the cold, for that is the nature of their business and failure to meet planned objectives would quickly have them out of work.
Having worked first hand on the recovery of many businesses after Sept. 11, I had the honor and privilege of dealing with the most dedicated staff of professionals in the country. These professionals were engaged in the Herculean efforts of rebuilding the torn and tattered remains of businesses that were fractured during the hours, days and months following the tragic events. They worked at a pace NEVER seen before in my entire 20 years of IT experience. Their efforts were amazing - from the front line technician servicing the customer to the procurement specialist working out of sight in the back office. Mr. Toigo's off-handed comment that these professionals will not deliver the best-of-breed in a time of crisis is so out of touch with the facts that I must question where he is acquiring his information. It's easy to talk about the practice of business continuity, it's another to actually live the hours and days associated with hundreds of recovery processes. Let us not diminish the good-natured deeds of hundreds of professionals working tirelessly to help others in a time of crisis merely because a friend is having trouble sustaining a business relationship with a vendor. Vendors are about selling their product. Those who think they have alternative goals are selling themselves short.
I also challenge Mr. Toigo on his views about vendors stepping up to the plate. In order to satisfy our company's business needs after Sept. 11, we were offered assistance by nearly every major vendor we do business with. In particular, I was touched to learn that one vendor, who shall go nameless, offered our company unlimited access to anything we needed that they provided, and it was my understanding that this offer came directly from the CEO. This vendor was one we competed with on a daily basis for business in some areas, and worked as a partner in others - and they delivered. These were not hollow gestures on the part of our vendors, these were genuine and generous offers of support.
Most vendors will do right by their customer when needed. But to reiterate a point, vendors are not responsible for a customer's business continuity plan. That responsibility should fall squarely on the shoulders of the CEO, CFO, CIO and any other XXOs including the board of directors. Vendors and resellers/integrators always walk a fine and tenuous line, but let's talk about the real issue: money. It's about which greedy salesperson is going to receive his or her compensation. Deals done at the corporate level between vendors and resellers are nice, but it's the deal done in the street that makes money. If vendor sales reps think they'll lose money from their pockets by doing business with resellers, they'll do whatever necessary to make sure that money stays put, even if it adversely affects a customer. I believe that a well-written compensation plan on the vendor's part would have potentially prevented this situation from occurring.
This was first published in September 2001