Hyperion Solutions found that having coexisting e-mail apps wasn't manageable. Here's how the firm standardized on Microsoft Exchange and adjusted its environment to accommodate the changes.
Hyperion Solutions Corp. began as a Lotus Notes shop, but acquired companies that used Microsoft Exchange 5.5 and Exchange 2000. Initially, the Santa Clara, CA–based business intelligence software company--with major offices in North America, Europe and Asia--pursued a coexistence strategy that allowed the various units to use their preferred e-mail platform while working behind the scenes to deliver what usually appeared to be seamless communications.
But by the fall of 2003, coexistence had grown to be more trouble than it was worth. "The coexistence approach was costly in terms of operational expenses and was subject to frequent failures," says Lou Tiseo, Hyperion's director of network architecture and engineering. Hyperion had to dedicate one full-time person to monitoring the e-mail environment to keep it running smoothly. Although it appeared to be a convenient way to absorb acquisitions without the pain of disrupting users and processes, in practice the strategy was proving to be increasingly untenable, especially as the company continued to grow.
|Exchange migration best practices|
Standardizing on Exchange
Hyperion decided to converge on a single e-mail system (Microsoft Exchange) for improved manageability, reliability and scalability, and to provide a consolidated platform for future messaging-based collaboration. The resulting consolidated messaging platform would also enable better e-mail security, especially for virus protection and spam control.
The firm quickly rejected alternatives to Exchange. Outsourcing e-mail would become quite expensive as the company continued to grow. Coexistence was too costly, and standardizing on Notes was rejected by users.
Standardizing on Microsoft Exchange proved to be an easy decision. "Most people were already familiar with Outlook," says Tiseo. "We looked at having Outlook clients work with the Notes back end. It works, but you lose some functionality." So the company decided on Exchange and Outlook.
There are two ways to pull off this kind of migration: a Big Bang approach in which you move everyone over in a weekend or incrementally, moving small groups of users over an extended period of time. "We don't recommend the Big Bang approach. Most companies are not prepared to deal with the negative user impact," says Scott Rodgers, a technology infrastructure practice director and Microsoft certified architect at Microsoft application integrator Avanade Inc. in Seattle.
Hyperion opted for a gradual, two-phased approach. The first phase centralized Exchange messaging storage groups at four regional hubs--California, Connecticut, the U.K. and Singapore--and migrated users to Exchange 2003. In the second phase, the company would build out Exchange messaging as a collaboration platform.
To execute the consolidation strategy, "we convened a team of about 25 senior-level engineers worldwide to hash out a unified approach," says Tiseo. The team had to address a number of issues, including how to handle Active Directory, how to set up the storage and how best to ensure high e-mail availability. They also had to contend with e-mail retention policies, mailbox sizing and security, and plan the actual user migration process.
Exchange requires Active Directory and "you have to deploy Active Directory first--get it up and stable--and only then start hooking Exchange to it," says Rodgers.
Tiseo said his team "didn't have much Exchange or Active Directory skills," so they engaged Hewlett-Packard (HP) Co. to run an intensive, week-long onsite Active Directory/Exchange training program.
The migration team spent the next three months working with HP to pin down the high-level design. In the end, Hyperion rolled out a single Active Directory domain globally to its four regional hubs and 30 branch sites. They also set up a management, monitoring and reporting infrastructure for Active Directory.
Before Hyperion began building out the new Exchange platform, Tiseo's team assembled a test lab. "We set up a lab environment that could mimic Exchange with 10,000 users. We would populate mailboxes, route mail, do everything we were intending to do," he explains. Today, the Hyperion Exchange system has 3,800 mailboxes; the lab was designed to support considerable growth.
The test lab also supported connections to Lotus Notes through Lotus Notes Connector, a Microsoft product used to move messages between Notes and Exchange; and Lightweight Directory Synchronization Utility (LDSU), an HP product that performed directory synchronization across all e-mail systems. The plan, however, called for eliminating Lotus Notes and ceasing the coexistence effort altogether.
"One of the most common mistakes I've seen is failing to do enough testing," says Rodgers. "Companies really need to build a staging environment and populate it with real production data. Then they can be sure that their design will work."
How many users the organization should migrate at one time depends on many factors, including the amount of data to be moved and the amount of user training required. As for data, Avanade's Rodgers recommends moving the previous 30 days' worth right away. The rest of the data can be moved to a second storage tier later.
The amount of handholding users require also impacts how many users can move at once. "You have to consider the increase in the number of help desk calls. We advise clients to expect a 10% increase in migration-related calls to the help desk," says Rodgers. After the initial flurry of calls, help desk activity falls back to normal levels until the next group is migrated.
After testing the migration process, Hyperion decided to move users in groups of approximately 100. "We tried moving 500 users once and it proved too difficult," says the firm's Tiseo. They targeted Notes users for the initial migration, followed by Exchange 5.5 users and Exchange 2000 users. While the migration was taking place, Hyperion maintained the coexistence environment using Lotus Notes Connector and LDSU to ensure nobody was stranded in the process.
Hyperion established new mailbox quotas as users were migrated. Notes users, due to the nature of Notes, had been accustomed to large mailbox sizes. "Notes handles mail differently on the back end, so large mailboxes aren't as much of a problem," says Tiseo. Hyperion's new quotas are 300MB for regular users and 800MB for VIP users. To a traditional Exchange shop this seems overly generous, admits Tiseo, but the migration team wanted to avoid culture shock. Hyperion also set an e-mail retention policy of 180 days.
Storage and backup
"Exchange works best with storage consolidated at regional data centers. There you also can have your high-availability clustered servers," says Rodgers. He usually recommends iSCSI with NAS arrays, such as those from Network Appliance Inc., which will do the job at a lower cost than a Fibre Channel (FC) SAN.
Hyperion, however, took the FC SAN route. "We used two EMC [Corp.] Clariion storage arrays with FC drives. It was redundant storage. We've never had a Clariion go down," says Tiseo. He realizes it's a more costly configuration than necessary, but the company likes the scalability of the Clariion and its reliability. In the future, Tiseo expects to add lower cost, second-tier storage.
Although Exchange doesn't require more storage, Hyperion's growth has driven an increase in Exchange storage. The company initially set up 3TB of mirrored storage on the two EMC Clariion CX700 arrays (1.5TB per array). It then uses Exchange's policy management to enforce retention policies. "We can set different retention policies for different groups. For instance, we can retain e-mail for VIPs for longer," says Tiseo. The current general retention policy is 180 days, after which e-mail is deleted.
For the future, Tiseo is exploring e-mail archiving in which messages would be automatically moved to lower cost SATA-based storage after 180 days. "This would free capacity on the production storage, and we wouldn't be spending a lot of money on high-cost storage to save e-mail," says Tiseo. The archiving effort would encompass not only e-mail, but other unstructured data. Archiving could remove 60% of the data currently on Hyperion's production storage. "If it were only for e-mail, it would be too costly," he adds.
For high availability, Hyperion runs active/passive Exchange clusters at each regional hub. Hyperion originally intended more complicated multiserver clusters, but found them too difficult, forcing the firm to go with single active/passive pairs for its clusters. "Exchange is not very cluster-aware, and it is not good at automated failback," says Tiseo. The company manually switches Exchange back after a failover.
In addition, the company configures the storage for increased availability by putting the database and logs in separate RAID groups. Tiseo also stripes RAID groups across enclosures. "That way, there is no single point of failure within the cabinet," he says.
Hyperion backs up Exchange using Symantec Corp.'s Veritas NetBackup with Exchange agents. Backups go to disk first and then to tape. Recovery parameters are seven days for deleted e-mail and 14 days for a deleted mailbox. Outlook clients are configured to archive mail to local .PST files every 90 days.
Upgrading Exchange versions
Hyperion uses Quest Software Inc.'s Exchange Migration Wizard to move data from older versions of Exchange to Exchange 2003. Hyperion also used products from Houston-based BindView Corp. (which was acquired by Symantec last year) to help define and manage security policies, and to demonstrate compliance with new and evolving regulations.
"Microsoft has tools with native capabilities, but we would need four or five different Microsoft tools to do what Quest and BindView did," says Tiseo. The Microsoft tools worked at a lower level, he adds, and were more powerful, but less efficient to use.
Avanade's Rodgers recommends that his customers use external tools. "The external tool vendors like Quest and Binary Tree [Inc.] specialize in this kind of work and can do it with the least impact on users," he says. Avanade itself has built a tool for internal use that works in conjunction with Quest. Using the Avanade/Quest toolset, Avanade consultants have migrated as many as 1,250 users in a day, an extremely high volume that Rogers claims you "can only do with our automation."
Tools like those from Binary Tree, BindView and Quest can't migrate users as quickly because they may still require manual intervention at various points in the process. For example, "you will have to update proxies manually," says Rodgers. That limits the amount of users you can migrate at once. Hyperion migrated users in groups of 100, which is considered a manageable number.
Hyperion also turned to third-party tools and managed services for spam and antivirus protection. "Prior to implementation, we wanted to minimize spam," says Tiseo. At that point, the company was dedicating 1.5 full-time employees just to monitor and manage incoming spam.
When he calculated what spam control was costing the company, Tiseo explored other options, primarily managed service providers. Eventually, Hyperion opted for FrontBridge Technologies Inc., which was subsequently acquired by Microsoft and became Microsoft Exchange Hosted Services. Hyperion is now blocking upwards of 18 million spam messages a month, which constitutes 98% of its spam.
For antivirus protection, the company turned to Symantec and Microsoft Antigen as part of a multilayered virus defense. It uses Symantec antivirus as a gateway and FrontBridge to scan for viruses. Antigen is used for local desktop scanning. "It has been so effective that we have been able to disable our junk mail filters," says Tiseo.
Only a few things caught Tiseo by surprise, and some of those couldn't have been predicted. The importance of e-mail in litigation, for example, wasn't on the radar screen of many companies in 2003 when the Hyperion migration began. "We didn't expect e-discovery," he says, but the company is now putting in a process for filtering and searching e-mail messages. At the same time, Tiseo's messaging team has left the Sarbanes-Oxley (SOX) controls to IT. "We don't have many SOX controls around Exchange," he adds.
The amount of data that must be archived and stored also surprised the team. "Our storage is growing at 80% a year. That tells me we need to address retention policies. We also need an architecture to restore messages fast; time to restore takes forever," says Tiseo. Hyperion is considering a second tier of storage, which it can use to facilitate e-mail archiving and e-discovery.
The migration to Exchange was essentially complete (except for new acquisitions) by December 2004. A year later, the firm embarked on the second part of the effort, messaging collaboration. In 2006, Hyperion began the deployment of Microsoft SharePoint.
The cost of the migration, from September 2003 through December 2004, was $1.2 million. Although Tiseo hasn't done a formal ROI, on the surface it appears to be money well spent. "It costs much more to manage multiple environments and coexistence than the Exchange platform. We've gone from 40 Notes servers to eight Exchange servers with high availability," he says.
The company has virtually eliminated spam, and has established a platform for Windows mobile access and enterprise-wide collaboration. Finally, the messaging team can now promise management that in any future acquisition, "we can give as many as 100 acquired employees their new Hyperion e-mail address on day one," says Tiseo. For a company driven by an acquisition strategy, that may be the biggest payback of all.