In previous tips in this series, we introduced the idea that implementing availability requires taking a layered approach, and then following that layered approach, we looked at good system administrative practices, backups, disks and storage (and why larger disks aren't always better), networking, and system's local environment.
The sixth level in the Availability Index (introduced in part 1) is Client Management. Since availability is an end-to-end proposition, it is vitally important that not only do the networks and servers work, but that there are clients who are available to access the services and data being provided by their servers. Just like servers, it's not just the client hardware that needs to work, but client-based applications, too. And, to take it one step further, it's not just that the applications need to work, but that they need to give correct answers.
In this tip, we're going to look at a finer point of client application availability, called value-based logic.
Did you ever reverse the number of shares and the share price of a stock purchase, perhaps when you entered it into Quicken or Microsoft Money? Imagine the potential consequences if an equities trader who wanted to buy 1000 shares of a stock at $100 reversed his numbers, and entered 100 shares at $1000. He would certainly find many eager sellers!
There was a famous real-life example of this sort of problem that occurred at a world-famous New York equities trading firm (who shall remain nameless) several years ago. There were two equities involved, NYNEX (symbol NYN; NYNEX is now part of Verizon,), one of the most actively traded stocks on Wall Street at the time, and the Nuveen Bond Fund (symbol NVN). Several years ago, a harried stock order clerk scribbled down an order for 10,000 shares of NYNEX, and gave it to a trader who misread the symbol as NVN. At the time, NVN typically didn't move 10,000 shares in a week. Nevertheless, the order was executed for NVN, rather than NYN. The damage to NVN's market caused by that single error took days of work to repair, and cost the firm over $300,000.
If the firm had employed a high quality trading application that compared the order to current market prices and sales trends, it would surely have identified a discrepancy. Before accepting the trade, it would verify that this rather unusual trade was, in fact, what the trader meant to enter. If it did so in a style that the trader was not accustomed to seeing, he would not just click "yes" and move on. Instead, the trader would stop for a second, and check to see that he had entered the information correctly. It would also give him a chance to correct his error.
A poor quality trading application would simply accept the order, and attempt to buy the shares.
Ultimately, the responsibility to deliver this functionality falls to the developers of the requirements document for the trading application's developers. Sadly, it's not the sort of requirement that usually makes it into requirements documents until the firm gets burned by such an incident. I am quite confident that the equity firm we discussed above makes that sort of value-based logic a fundamental requirement of any application that they develop in house.
Copyright 2003 Evan Marcus
Evan L. Marcus is data availability maven for Veritas Software.