E-mail: It's worse than you think


This article can also be found in the Premium Editorial Download "Storage magazine: How storage managers can survive e-mail archiving."

Download it now to read this article plus other related content.

It's official: Nothing beats e-mail. According to a 2003 study by Meta Group, 80% of businesspeople say e-mail is more valuable to them than the telephone. And just like phone networks, e-mail systems experience enormous strains behind the scenes that may cause problems--but users don't want to hear about it. They just go berserk when their e-mail systems don't work.

Requires Free Membership to View

No Tampering, Please
Many people assume that Rule 17 (specifically subsection 17a-4) refers to write once, read many (WORM) optical discs. After all, when you burn it on WORM optical media, nobody can change it short of destroying it. But in a SEC Interpretation published on May 7, 2003, the Commission made it clear once and for all that "17a-4 does not require that a particular type of technology or method be used to achieve the non-rewriteable and non-erasable requirement." That is, you can use any "electronic storage system that prevents the overwriting, erasing or otherwise altering of a record during its required retention period through the use of integrated hardware and software control codes."
This ruling instantly legitimized a number of magnetic WORM solutions already on the market. These give you the speed of a near-line magnetic disk array, plus special logic and software to make tampering more or less impossible. The best known is EMC's Centera Content Addressed Storage system. Introduced last year, this WORM magnetic disk array that recently got a makeover in a Compliance Edition that claims to meet the letter of the SEC Interpretation. Network Appliance offers similar solutions via its SnapLock software, which can be used in conjunction with several of the company's NearStore arrays.
WORM takes other forms. StorageTek offers a WORM tape solution, as does Sony. And JVC specializes in DVD-R solutions, such as the MC-8000 series. But with rewritable media, the secret is in software that encodes data to prevent tampering--along with policies and procedures to make the process verifiable. Bill Peldzus, a consultant with GlassHouse Technologies, in Framingham, MA, says that you need to have "some type of manual certification that the way you are storing this and the way it's presented is the fact its original form. So there are manual processes along with technological processes that basically verify the integrity of the data."

The stresses on these systems are formidable. Corporate misbehavior has resulted in complex new regulations for e-mail retention--and has forced IT managers to take existing regulations seriously. Meanwhile, spam is flooding enterprises, not just home user mailboxes. Like a Malthusian curve, e-mail storage requirements just keep climbing, with each enterprise user now eating somewhere between 5MB and 10MB in e-mail per day, a volume expected to double by 2006, according to IDC.

"Everyone is having trouble managing the growth in e-mail data stores," says Carolyn DiCenzo, a VP of research for Gartner Inc. "And in some industries, e-mails are also important records of transactions. So people don't just read e-mails and delete them anymore; they actually keep them in their mailboxes for much longer." Larger e-mail data stores make backup and recovery more difficult, says DiCenzo--especially since people now expect nothing less than 24/7 availability.

Unlike large enterprise transaction databases, e-mail databases can't increase without incurring major performance penalties. So archiving old e-mail is essential; it represents a different process than basic backup protection, which can be complex in itself, depending on how coarse (coarse meaning it involves the entire data store) or granular (occurring mailbox by mailbox) the restore needs to be.

Ratcheting requirements even higher, some enterprises now recognize that e-mail stores serve as de facto repositories for intellectual property. As a result, they're beginning to treat messages as just another data type in a document or content management system, making quick search and retrieval a requirement.

To slay the e-mail hydra--and eliminate concerns over backup and recovery, scalability, archiving, spam control and content management--vendors have forged new weapons, and enterprises are establishing new lines of attack. All involve software or hardware solutions used in conjunction with the major e-mail servers: Lotus Domino and Microsoft Exchange. Of course, the servers have also evolved. Both Exchange 2000 and Domino Server Version 6 will store only one copy of an attachment, even if that file is attached to multiple messages. But a complete e-mail management solution, especially for companies with special regulatory needs, requires additional software. And in almost every case, IT must also shoulder the burden of persuading users to change their e-mail habits.

The monster that ate my server
In countless organizations, IT managers express frustration that something as mundane as e-mail seems to take so much of their time. "I spend at least two hours every day dealing with Exchange in some way," says Mike Wolf, manager of technical services for LifeCare Assurance, in Woodland Hills, CA, a fast-growing insurance administration company that specializes in long-term care services. "Our information store went down a week ago, and I was here until 4:00 in the morning."

Wolf's overriding concern is the same basic bugbear that everyone is dealing with: a ballooning e-mail store. Though the company has 80,000 to 90,000 agents in the field, Wolf has only 350 Exchange 2000 mailboxes for employees of the company's corporate offices to worry about. Yet the 60GB of disk Wolf installed two years ago--figuring it would last a long time--is already hitting the wall. "We're at the 22GB [mark] now," he says. "In Exchange, you need [to] double your disk space, so at 30GB we'll be over our limit."

Although 30GB may sound like a pittance, the issue isn't simply space, but also backup and recovery overhead. "Our current recovery strategy is a nightly full tape backup on the data store and an individual mailbox backup for individual message restores,"says Wolf. "The reason I do it that way is because this is an insurance administration company, and one e-mail can save or or ruin the day." If one of LifeCare's VPs wants an e-mail from two years ago, says Wolf, trying to dig that out of a full data store backup is a non-starter, so he restores from the "bricklayer" backup, as the mailbox-level backup is called. But even that is time consuming.

Laying down the law with size limits on end-user mailboxes is one obvious way to reduce such overhead. But here's where the Platonic ideals of best practices wither in the cold light of day. "We look at a lot of case studies on successful Exchange implementations and the policies and procedures for e-mail," says Wolf. "But we haven't found anything that fits LifeCare yet because people work directly out of their inbox."

Wolf ruefully admits that a culture of using the e-mail system for what amounts to document management stems from a lack of foresight on IT's part, but he feels that it's too late for draconian cutbacks on mailbox sizes. "If we were creating a brand-new environment, we would make it that way from the start, and the user would never know the difference," he says. "The problem I've got now is several hundred users who do know the difference. We let it be a free-for-all, and it's going to continue to be that way because they're used to it being that way. So our only choice is to add storage to our Exchange server and rethink our Exchange environment."

That rethinking has coalesced into a plan for next year to move to Exchange 2003 and to set up a local ActiveX cluster sharing the IP SAN device for the information store and the data for Exchange, says Wolf, who eventually hopes to have a three-node cluster with one remote in a redundant hot site facility. "By doing it that way, you get the redundancy you're looking for and you get the storage capacity you're looking for, because with IP, SAN storage is almost unlimited. You can resize those partitions almost on the fly."

The solution Wolf is considering is a storage concentrator from IP storage vendor StoneFly Networks, in San Diego, CA, which circumvents the usual objection to e-mail SANs--too much expense for an application that's less than mission-critical--by using Gigabit Ethernet and iSCSI-based storage (see "Is an e-mail SAN right for you?"). But there's another effort Wolf realizes he needs to make: Soon, he plans to roll out an archiving program that requires users to retain old messages in archive folders or lose those messages for good.

This was first published in August 2003

There are Comments. Add yours.

TIP: Want to include a code block in your comment? Use <pre> or <code> tags around the desired text. Ex: <code>insert code</code>

REGISTER or login:

Forgot Password?
By submitting you agree to receive email from TechTarget and its partners. If you reside outside of the United States, you consent to having your personal data transferred to and processed in the United States. Privacy
Sort by: OldestNewest

Forgot Password?

No problem! Submit your e-mail address below. We'll send you an email containing your password.

Your password has been sent to: