Move data fast

High on Thomas Fenady's priority list at Activision Publishing Inc. has been improving file-transfer speeds for traffic over the WAN. Fenady is senior director of IT at the Santa Monica, CA, video entertainment company, which publishes coveted titles such as Guitar Hero, the game that gives everyone a chance to rock like Jimi Hendrix.

While the main data center is at the company's California headquarters, Activision's IT users are spread across 67 offices on four continents. In Fenady's favorite example of how an international video entertainment firm can pummel its network, the Dublin, Ireland-based testers and designers working on Call of Duty routinely generate approximately 200GB of data each day.

That's because the video game, which lets players fight World War II armies, is published in 16 languages.

"We'd be sending an 8gig file 16 times," explains Fenady. "We were getting 3megs per second on an E3 line, and we were able to tune that to get about 8megs per second." Then Activision joined a growing list of companies using WAN optimization products as a way of improving business continuity and application performance. With Riverbed Technology Inc.'s Steelhead appliance, says Fenady, "we started getting about 34megs per second. We were filling up the pipe."

Activision has about 7,000 users (a number that tripled recently after the firm merged with Vivendi Games to form Activision Blizzard). His users "don't see a jump from 8megs to 34megs," says Fenady. "What they see is a game-build transfer that could take eight hours before and now takes 30 minutes."

The need to improve application response times is just one of the factors driving the WAN optimization market, which grew out of the old wide-area file services (WAFS) sector. Interest is currently being stoked by the following IT trends, each of which is inexorably linked to both network and storage systems.

• Storing remote-office data in the main data center



• Increased use of collaborative applications



• Business continuity: Failing over from one data center to another means moving multigigabyte chunks of data across a WAN

"The standard approach [to accommodating increased WAN use] is to overprovision the amount of bandwidth you need," he says. But purchasing more bandwidth can be very expensive. According to Whiteley, a company with an international E3 line going into Bangalore, India, for example, might pay up to $40,000 each month for a private link. But WAN optimization products, for a fraction of that cost, reduce bandwidth costs by reserving bandwidth for priority traffic and using data-reduction technologies.

"The bad news is that [WAN optimization] is still a nascent technology, so it comes with limitations," says Whiteley. These products require plenty of initial testing and careful deployment to avoid problems with reliability and scalability, he adds. "You can't just pick the vendor you have the best relationship with or the one with the best price." (See "What to test," below.)

				What to test
				Before getting sidetracked by throughput, beware of some of the potential pitfalls of buying a WAN optimization product that you haven't tested thoroughly. Scalability: As your WAN traffic grows, how scalable is the optimization appliance? How costly and difficult is it to scale? How many protocols does the appliance handle? Security: How does the vendor secure its appliance? Does your WAN optimizer need to accelerate Secure Sockets Layer (SSL)-encrypted traffic? Backing up the box: Do you need a redundant device in your main data center? What happens if a box fails? Visibility: What level of performance is the WAN optimization box delivering during different times of day, for different applications and to how many users? In other words, how is the remote-office user experience?

Deployment options
At Activision Publishing, one of Fenady's initial concerns turned out to be a non-issue. "The Riverbed [Steelhead appliances] moved everything onto proprietary ports," says Fenady, while other products "kept everything on native ports." With a Steelhead appliance at both ends of the WAN (he started with one at the California headquarters and another in Dublin, Ireland), Fenady discovered that his traffic was now moved to a single, proprietary port. "So then you need NetFlow [the proprietary network protocol developed by Cisco Systems Inc. for collecting traffic information]," he says. By enabling NetFlow on the Riverbed appliance, Fenady sees what he needs to. Primarily, Fenady wants up-to-date numbers related to bandwidth utilization; for troubleshooting, he likes to dig deeper into traceroutes, retransmits and cyclic redundancy checks.

Getting the most out of a WAN optimization product, says Fenady, comes down to deciding how to deploy it. The leading WAN optimization vendors support two classes of deployment: in-path and out-of-path (see "In-path vs. Out-of-path WAN optimization," below).

				In-path vs. Out-of-path WAN optimization
				WAN optimization customers typically have two choices for deployment: in-path or out-of path. Out-of-path deployments are more technically challenging, and while there are no pre-set conditions for determining which deployment method is better for a single IT shop, out-of-path deployments are sometimes the first choice of companies that choose to accelerate particular subsets of their traffic--applications or routes they know are tying up bandwidth. An out-of-path deployment is also a likely choice for data centers where there's no obvious single point at which in-path appliances should be deployed. When deploying out-of-path, users decide on protocols: Policy-based routing (PBR) or the more common Web Cache Communications Protocol (WCCP). WCCP does consume some memory and CPU cycles on the router or switch. For some, that's a non-issue. With older-series routers, that might be a problem. With in-path deployments, things are simpler. There are no routers to configure, and no memory to add. In a single-path deployment, the NICs are "fail-to-wire," explains John Martin, VP of product management at Riverbed Technology Inc. "In other words, if there's a software or hardware failure, the NICs become a wire and the traffic passes straight through as if the device wasn't there," he says. Approximately 70% of Riverbed customers deploy their appliances in-path, he adds. Examples of an in-path deployment and out-of-path deployment are shown here. In an in-path deployment, an appliance is inserted between the WAN router and the Ethernet switch on the LAN side of the network. Out-of-path deployment of a WAN optimization appliance with the Web Cache Communications Protocol (WCCP). Diagram source: Silver Peak

In-path deployments require placing the WAN device between the switch and the router. The appliance is inserted in-line between the WAN router and the Ethernet switch on the LAN side of the network. Out-of-path deployments require reconfiguring routers, and appliances are deployed in each office of a distributed enterprise network, typically behind the WAN router. Fenady chose to deploy his Riverbed appliance in-path, he says, "since the Riverbed has a failover pass-through. If the unit dies, there's no network interruption; we tested this extensively."

The leading WAN optimization vendors support both classes of deployment, and customers choose their deployment based on infrastructure, preferences, traffic flow, and the number of users and branch offices. When Cisco first introduced its WAN optimizer product, it was offered in only out-of-path deployments. Mark Weiner, Cisco's director of market management for data center solutions, says the company's Wide Area Application Services (WAAS) software (that's loaded onto an appliance) is now offered in-path as well in response to customer demand.

Jeremy Gill is CIO at Pittsburgh-based Michael Baker Corp., a survey and engineering firm with 4,200 employees and a network that supports 50 domestic and eight international offices. The company, whose work includes flood-plain mapping for federal agencies like FEMA, ran into a problem after it deployed the ProjectWise content management app. At the same time, Gill was concentrating on consolidating 15TB of corporate data to a main site. "As that data center grew and we consolidated file servers, we realized we were going to have to increase our WAN acceleration," he says. Gill had a "ton of Cisco ISR router lines," so choosing Cisco's WAN optimization product and deploying it out-of-path made sense to him.

Gill is an IT veteran who was never crazy about the WAFS products of yesteryear. "To be honest," he says, "I'm not a fan of file acceleration. If there's a piece of hardware that will do it, I'd rather do it at the hardware level." In the end, Gill wound up with the kind of WAN optimization ROI he was looking for: $4,000 in monthly bandwidth savings, increased remote-worker billable hours and centralized storage.

Early on, Gill was unhappy with Cisco's reporting tool, but he says the company has made strides in recent releases. "There are more out-of-the-box reports now compared to us having to pull the data and run our own manipulation of it," he says. "That was taking 20 hours a month." But the numbers, he notes, are impressive. A six-month snapshot showed ProjectWise accelerated by 60%. Gill is also looking forward to the improved video streaming in Cisco's most recent release, he says. In the older version, all 20 users in a remote office would receive their own video stream, sent across the WAN from the media server in the company's main data center. A newer version of WAAS pulls only one stream from the media server, which is then sent to the 20 users from the WAAS device in a remote office. In the case of small WAN links, such as T1, which a lot of Gill's remote offices use, "it can't support a stream of 20 users at 384K, but it can easily support one," explains Gill. "That will help us significantly. We have folks in remote offices, logging into Web-based meetings, that kind of break the bandwidth when it comes to video."

Jeff Post is network manager at EDC Inc., an international nonprofit that does work in 35 countries, and has programs ranging from health care to education and grassroots economic initiatives. For Post, that means heavy WAN use, especially between the nonprofit's three regional offices in Newton, MA; Washington, D.C.; and New York City.

Post became a Silver Peak Systems Inc. customer in February when EDC began a data center consolidation project. "Our need [for a WAN product] came when we outgrew our space, power and HVAC supply," says Post.

Reducing WAN traffic
Alan Saldich, Riverbed's VP of product marketing and alliances, says many customers, like Activision's Fenady, use Riverbed to accelerate or dedupe their SnapMirror traffic. "Our dedupe algorithms are much more granular; what looks like a new block to SnapMirror often looks to us like repeat bytes. Whatever you use--SnapMirror, Double-Take [from Double-Take Software Inc.]--once you start the replication job we make it go as fast as possible," he says.

EDC is mostly a Cisco shop, but Post says he was open to various WAN vendors when he began looking at WAN optimization products; Post says the Cisco product he tested didn't work so well with the 200 or so Macs in his shop. In the end, he chose Silver Peak and says "they didn't accelerate my Mac traffic either, but they didn't break it"; Post also says Silver Peak told him more Mac capabilities are on its roadmap.

Damon Ennis, Silver Peak's VP of customer support and product management, says Silver Peak's Mac acceleration was limited at the time to a certain feature. "We were able to accelerate file copies, but we weren't able to accelerate what we call directory browsing specifically for Apple Macs; that has since been remedied," he explains. Cisco's Weiner says the firm has plenty of customers using Cisco's WAAS for Macs; he noted a recent enhancement to Cisco's WAAS, an HTTP adapter that aids Mac traffic.

Post chose an out-of-path deployment, and reconfigured his Cisco Catalyst 3750 series switches for policy-based routing so they would send data to the Silver Peak appliances. "We didn't consider doing it in-path," he explains. "We have tried, as much as possible, to not have a single point of failure in our network design. We have two switches in each location handling the routing of the data. I feel the out-of-path deployment offers more flexibility."

Post chose to implement policy-based routing (PBR) redirection so the appliance intercepts only those packets that have been redirected to it. It accelerates traffic flows that match its Access Control Lists (ACLs); all other traffic passes through the appliance unmodified. But Post's policy-based routing rules were too broad at the outset, which posed a problem. "Originally, we said we wanted everything to go over it. Well, if you send everything, you wind up sending a lot of things you never intended to, such as print jobs from remote offices," he says. "Now we just lock down the IP ranges that [print jobs would send to]."

				New WAN optimization features and products
				In the next few months, you're likely to hear plenty of new product news from WAN optimization vendors. Here are some things to look for, but you'll need to determine where they fall on your priority list. Broader lineup of appliances: from low-cost entry models to high-end clusters Additional protocol-acceleration options Bigger boxes to handle larger data streams Acceleration software on-demand, accessed through a VPN for remote users Enhanced deduplication Acceleration of Secure Sockets Layer (SSL) traffic across the WAN Speedier delivery of voice and video traffic

Product competition heats up
As the WAN optimization market has matured over the last four years, vendors have been competing on features such as advanced compression techniques and traffic management, including load balancing and protocol-specific optimization for Web-based applications. Blue Coat Systems Inc., with a proxy-based architecture, sells itself as the company that combines security and acceleration--keeping out the bad and speeding up the good.

Riverbed boasts the ability to decrypt Secure Sockets Layer (SSL)-encrypted traffic, and Forrester ranked it highest overall last year for optimization techniques. Silver Peak, which received high marks for its scalability in Forrester's Q3 2007 market report, is making that a cornerstone of its product pitch, and company executives say their roadmap includes larger appliances and the ability to handle an even larger data stream.

However, users are discovering some pockets of trouble when it comes to implementing WAN optimization products and scalability tops the list, says Forrester's Whiteley. "Does it process enough TCP flows? Does it have enough on-board disk capacity? It's not just about megabits per second," he says. In general, WAN optimization products scale out (the number of devices that can be interlinked) and scale up (the amount of optimized throughput each appliance can process).

"You should test solutions that support the right throughput on the WAN connection with all of the optimization techniques turned on," says Forrester's Whiteley.

Another problem can be reliability or backing up the box, he says. "Some companies are getting burned by the fact that there aren't modular components; they aren't hot-swappable," he says. In one example, a Toronto company with Bangalore, India-based developers discovered it had reached 80% WAN bandwidth utilization, "which is almost catastrophic" says Whiteley. The company installed WAN optimization appliances on either end for reduction in overall traffic and cut its WAN bandwidth utilization to approximately 40%. That worked out great ... for a while.

"Over time, that utilization crept back up to 60%," says Whiteley. The "creeping" of utilization--in this case, from 40% to 60%--is very common, he adds.

But without the WAN appliance, the firm is operating at 120% of link capacity and must be prepared in case the appliance fails by installing "high-availability pairs or trying to get solutions with higher reliability like modular hardware, in-service upgrades and hot-swappable components," says Whiteley. (That's the strategy Gill at Michael Baker Corp. applied.) Riverbed's Saldich says users don't need to back up every box with another. "Let's say you have 100 branches," he says, and one big box in the data center. "If the one in the data center is deployed serially or in parallel, that's a pretty reliable deployment," adds Saldich.

Prices for WAN optimization products vary depending on network size and the number of sites. A good starting point is approximately $50,000 for a major location such as your data center, and $15,000 to $20,000 for branch sites (pricing for remote locations vary).

Brian Babineau, senior analyst at Enterprise Strategy Group, Milford, MA, predicts that WAN optimization products will be priority budget items for IT shops this year and next. The cost of buying more bandwidth, at a time when IT shops are under increasing pressure to restrict expenses, makes WAN acceleration products easy to rationalize, he explains.

"We've talked to a lot of people who have purchased WAN optimization solutions and we haven't had one of them say 'Hey, this stuff didn't work out for us,''' says Babineau. "That's pretty unusual. A quick ROI is what counts right now."

				Glossary of routing, network and security terms
				As with most IT product categories, WAN optimization has its own set of terms. Here are some key ones that you should be familiar with as you evaluate WAN optimization devices: T3 and E3 telecommunications services: T3 service provides transmission at a rate of 44.736Mb/sec, while E3, the European equivalent, is rated at 34.386Mb/sec. Policy-based Routing (PBR): A technique used to redirect traffic to WAN appliances using Access Control Lists (ACLs) set by the user. Secure Sockets Layer (SSL): A cryptographic protocol that provides security for data transfers. TCP flows: The Transmission Control Protocol provides in-order delivery of a stream of bytes. Web Cache Communications Protocol (WCCP): A Cisco-developed, content-routing protocol that can redirect traffic flows and perform load balancing, scaling and fault tolerance. The protocol can also localize Web traffic patterns in the network, which enables content requests to be fulfilled locally and reduces transmission costs.