Published: 11 Jul 2005
Storage managers must deal with stricter government regulations and rapidly escalating e-mail stores. There are many e-mail archiving programs available, but finding the one that best meets your company's needs is the key. p>
E-mail archiving applications
Click here for a comparison table about e-mail archiving applications (PDF).
With stricter government regulations and rapidly escalating e-mail stores, storage managers are caught between a rock and a hard place. E-mail archiving tools are designed to automate the process of what to save and what to toss. But to keep costs down, these tools must do more than just move messages from one type of media to another--they must be able to examine, classify and manage the content of the messages so they can satisfy internal business policies and government auditors.
There are more than 30 software tools on the market that enable organizations to manage archived e-mail messages. Finding a product is less of an issue than choosing the right one for your company. High on any e-mail features checklist should be ease of use, comprehensiveness, TCO, scalability, and indexing and search functions.
What regulators look for
Michael Casey, vice president of data policy and ILM services at consulting firm Contoural, offers these tips to organizations that need to create e-mail archiving policies that comply with industry regulations.
Save messages for a defined period. All regulations specify a minimum retention period, usually three years. In the event an investigation occurs, messages will need to be retained longer, so verify that an archiving tool can suppress the deletion of messages if necessary.
Keep records safe and accessible. Regulators expect users to keep records safe from loss, damage or misuse.
Ensure data integrity. Storage administrators must protect records from alteration, damage or deletion while in the archive. Write once, read many (WORM) media or the archiving software's encryption and auditing features may be used.
Maintain confidentiality. An individual's information must be kept secure with administrative safeguards in place.
Availability. Courts and regulators continue to raise the bar in terms of accessing data. Auditors now expect the information in minutes or hours, not days or weeks, as in the past.
Before evaluating e-mail archiving products, Contoural Inc., a Los Altos, CA-based storage consulting company, recommends users first establish corporate policies on who needs to handle e-mail archiving. In many organizations, the management of archived e-mail messages remains a grey area; the responsibility may lie with groups ranging from the e-mail team, document management team, storage team or systems admin to a team cobbled together from all of the separate IT silos.
The next step is to identify the product features your company needs. Key product features include the ability to:
- Set policies and rules to manage the archival and retrieval of e-mail messages.
- Search and classify the content of e-mail messages, including attachments.
- Minimize the e-mail archiving tool's impact on the e-mail server's performance.
- Manage disparate e-mail archive stores, such as on PCs or laptops.
- Produce reports that track capacity growth and satisfy auditors' requests.
Policies that comply
Critical to any e-mail archiving package is its ability to set policies and procedures that meet business and government requirements. Products such as Veritas Software Corp.'s Enterprise Vault (acquired from KVS Inc.) and iLumin Software Services' Assentor Enterprise Suite have wizards to get them up and running quickly.
Mary Kay Roberto, senior vice president and general manager, North America KVS, a business unit of Veritas, notes that Enterprise Vault doesn't offer its users the option to choose a wizard that, if selected, would allow the organization to declare itself compliant with a specific regulation. Rather, users are urged to interpret how specific regulations apply to them and then set policies accordingly. She suggests organizations set up an internal compliance committee to determine what policies should be put into place. She further recommends companies work with an outside counsel specifically trained to set up e-mail rules for their industry. Navigating the regulatory shoals is complicated and expensive because courts interpret and apply existing laws and rules differently. Users may also need to use a specialized appliance to satisfy their e-mail archiving requirements (see E-mail archiving accessories).
For more mature regulations, it's easier to apply specific retention policies to archived e-mail messages. For example, HIPAA regulations are reasonably well defined, but other regulations are not so clear-cut. One e-mail archiving company spokesman has even called Sarbanes-Oxley "a slippery devil" (see "What regulators look for", this page).
It may be tempting to set a policy to delete all e-mail messages after 60 or 90 days, believing the problem solved. Michael Casey, vice president of data policy and ILM services at Contoural, advises clients not to bet on that option. Casey says courts care little about the difficulty and cost of discovery. And if the company policy is to delete all e-mails after a certain time, users will stash the e-mails they want to keep on CDs, local disk drives, USB flash drives, floppy diskettes and even print copies tucked away in file drawers.
The legal murkiness surrounding what to archive, coupled with the potentially unpleasant consequences of deleting everything, push many storage admins toward keeping it all. That's the approach John Hegner, vice president of technology services at Liberty Medical Supply, Port St. Lucie, FL, took. While the firm has content filters to block e-mails, Liberty's Exchange server with its 1,000 mailboxes still processes approximately 50,000 e-mails a day that must be archived.
Michael Sherwood, CIO for the City of Oceanside, CA, archives every e-mail message in a central location, but had a different problem. Oceanside, like many state, city and county government agencies, falls under the Freedom of Information Act. A component of this law allows citizens to request a copy of any sent or received e-mail for the past two years, so all e-mails must be available to be searched and retrieved.
Keeping all of these e-mails creates its own set of problems. While Sherwood has filters to catch spam containing pornography or vulgar language, he finds spammers getting more innovative in their e-mail text and subject lines. As a result, more objectionable spam bypasses his filters and ends up getting archived.
E-mail archiving accessories
For some e-mail archiving applications, you'll need a specialized appliance or software that's specifically tailored to your archiving requirements. Some options to consider include:
Write once, ready many (WORM) media. EMC's Centera, Network Appliance's SnapLock disk systems, Plasmon's UDO and StorageTek's 9840 VolSafe tapes all offer ways to store and protect e-mail messages for years or decades in unalterable formats.
Centralized database. While most e-mail archiving products maintain and manage only their own database, CommVault's QiNetix suite allows users to integrate, catalog and search data across their backup, e-mail, storage management and migration products.
Protecting desktop e-mail. Companies that need to protect .pst files residing on desktops and laptops should consider products such as LinkPro's PowerSync, which lets users sync .pst files between local e-mail files and central stores.
Storage scalability. Administrators can easily underestimate future e-mail growth, both in terms of message volume and the size of attachments. Midrange arrays like Hitachi Data Systems' 9500 or ATA products like Nexsan's ATABeast allow users to scale their back-end storage at price points that meet different e-mail archiving performance and availability requirements.
Data protection. Just because e-mails are archived doesn't mean they're protected. Don't forget to apply an appropriate level of backup and recovery for your e-mail archive application and stored messages.
Features and functions
Most e-mail products give admins the ability to apply policies by message, mailbox, user, group or globally. The law firm of Willkie Farr & Gallagher in New York, for example, has established that e-mails are the property of its clients--not of the firm. It sets up e-mail archive policies based upon the retention requirements of clients.
The ability to search e-mail archives by key words or phrases is a critical feature in keeping users happy and auditors satisfied. Archiving tools speed and improve the search process because they create their own indexes separate from the e-mail server. Equally important, the archiving software breaks apart e-mail attachments and indexes the content of these documents, allowing searches and discoveries to be done on these as well. For instance, Veritas' Enterprise Vault supports more than 250 types of attachments, and can break open and index documents in a variety of formats, including .pdf, .xls and .ocr.
Users need to be conscious of their search and discovery requirements if they want a single product to search their primary e-mail storage and archived e-mail. Some products index archived messages and messages still on the e-mail server, so all messages across the e-mail infrastructure can be searched. Companies may also find it advantageous to buy backup and e-mail archiving software from the same vendor. But Denise Reier, a vice president at EMC/Legato, was forthright in saying that her firm's EmailXtender and NetWorker software are independent products and that no particular benefits would be derived by purchasing both from EMC/Legato, save holding one vendor accountable for support.
E-mail administrators will also find improved performance on their e-mail server once the appropriate e-mails are archived. Moving this data off the primary e-mail server to an archive improves server efficiency in at least two ways. First, it reduces system overhead by eliminating the need for the e-mail database to store and manage all of the messages. Second, it reduces the backup and recovery window of the e-mail server because fewer messages reside on the server.
Not all archiving features reduce the server load. For users looking to archive all messages in real-time as they enter their system, most e-mail archiving products require users to take advantage of features native to the e-mail server product, such as message journaling on an Exchange server. Enabling this feature causes the message server to capture and send a copy of each e-mail message to the archive server, but it also adds a load of anywhere between 5% to 20% to the e-mail server. In some instances, vendors recommend not using this feature, especially if budgets are tight and you're not in an industry scrutinized by regulators. One e-mail consultancy recommends archiving messages once they're seven days old. Users will have deleted most of the messages they don't want or need by that point, and the number of messages that need to be archived will be reduced.
Another archiving feature detaches e-mail attachments from the primary e-mail store as they come into the system, leaving a link that lets a user open the attachment. This feature is especially appealing to dial-up users who don't want to waste precious time downloading large files. But it should be implemented carefully, as the link may not reveal sufficient information about the contents of the e-mail, possibly causing users to miss important e-mail attachments.
For some businesses, notably financial services, instant messaging (IM) systems may require the same kind of archiving and search-and-retrieval capabilities. While many e-mail archiving applications also support IM, make sure your vendor supports the particular IM product your company uses.
Despite the existence of central mailboxes, some users still elect to keep thousands of e-mails on their PCs, stored in local files such as Exchange .pst files. To address this potential liability, tools such as Legato EmailXtender offer administrators the ability to crawl the LAN and archive older messages from locally stored .pst files.
Implementation and maintenance
Once an e-mail archiving package makes it in the door, administrators begin managing the product. While administrators should always proceed cautiously with any new product install, this is especially true for an e-mail archiving tool because e-mail has become a critical application for many companies.
Nearly every vendor has had some missteps associated with early releases of their products. C2C Systems Ltd., Springfield, MA, for example, reported that early versions of its Archive One product didn't include a throttling feature to regulate archiving activity. Without throttling, archiving a year's worth of old messages could consume so many CPU cycles that e-mail servers would have trouble processing new messages. Archive One now includes a setting that limits the amount of e-mail it can archive at any one time.
Vendors recommend that users take a few preparatory steps when implementing e-mail tools:
- Schedule the initial archiving processes to run during off hours or on weekends when e-mail activity is low.
- Don't archive all old e-mails at once. Rather, use policies within the tool that limit the amount of e-mail that's archived during any period of time. For instance, if your business needs to retain e-mails for five years, set your policy to only archive e-mails up to five years old.
- Start archiving messages within less-visible user folders, such as the sent and deleted folders, rather than immediately archiving in-box messages.
- Begin an e-mail archiving effort with a test involving a small set of users and then gradually add more users.
The reporting capabilities of archiving applications should also be addressed during the implementation period. Reports generally fall into two categories, capacity and compliance. On the capacity side, reports should provide basic information, such as the number of e-mails archived daily, the total number of e-mails archived, the amount of archived storage and the number of e-mails scheduled for deletion. For compliance, auditors tend to be most interested in documentation of user and password management, backups and restores of the e-mail archive, and version management of the e-mail archiving software.
Once an e-mail archiving system is fully implemented, administrators report few, if any, issues other than routinely applying patches and upgrades as they become available. It's important to note that patches or upgrades to e-mail systems may affect e-mail archiving programs. The City of Oceanside's Sherwood found that because of the frequency of security patches coming out of Microsoft, he occasionally had to wait until the patch was certified by the Enterprise Vault software before he could apply it to his Exchange server.
However, the biggest issues surrounding e-mail archiving are ever-evolving governmental regulations. Selecting an e-mail archiving product that's a good fit with your storage environment is the easy part; interpreting and complying with regulations is a lot tougher. In the end, creating an effective and workable e-mail archiving process is a company-wide process.