Measuring End-to-End Applications Performance

By David Strom  |  Posted 2009-04-09

The tide is turning on how enterprises deliver applications, and the change is making it hard for some IT shops to adapt. The problem is that applications aren’t just about transactions anymore: They have far-reaching consequences of delivering content across a wide swath of Internet and local network infrastructure. Monitoring that infrastructure with tools of the past isn’t easy, as these tools tend to focus on a single area or layer in the protocol stack.

Complicating things is the fact that keeping your applications infrastructure running smoothly and delivering the best possible overall performance for your users can require installing multiple products, such as WAN application accelerators, reverse proxy servers, bandwidth monitors and load balancers. Having staff with the skills necessary to maintain these products—and to understand when they have intermittent failures or experience congestion—is a challenge.

“As more services—particularly real-time communications applications—are delivered across the network infrastructure, it’s critically important to monitor and manage the overall performance of the network,” says Sherrie Littlejohn, executive vice president of Network Services & Operations at Wells Fargo in San Francisco. This may mean more work in terms of what to monitor. “In addition to network latency, we measure jitter, packet drop and other performance characteristics,” she says.

As firms deliver complex applications that involve Web servers, distributed databases and other Web 2.0 devices, the situation is going to get even more complex because parts of this infrastructure lie outside the traditional IT sphere of control.

“Although we do trend analysis on our network’s utilization, we like to check on our network during shift changes or on Mondays because that’s when our call volumes are highest,” says Bart Waress, the IT director at Home Buyers Warranty in Denver. He uses monitoring tools from Cisco and Qwest to keep track of end-to-end performance.

“We haven’t had to upgrade our bandwidth since the move to MPLS [Multiprotocol Label Switching] two years ago,” he says, “even with the increasing use of cloud-based applications such as and the move to online customer care. Given the state of the economy, I would rather spend $40,000 to upgrade our applications than buy a load balancer right now.”

“We’ve been able to alleviate the need to upgrade our Internet/WAN connections and get significant bandwidth savings by deploying Cisco’s WAN optimization technology and using SolarWinds’ Orion package for monitoring,” says Jeremy Gill, the vice president of IT for Michael Baker in Moon Township, Pa. “In our first six months, we realized a nearly 11 percent overall traffic reduction over our WAN, and with one of our production applications, we saw a more than 50 percent traffic reduction. IT managers need to look beyond simple load balancing and look at truly optimizing which network layer makes the most sense for their applications performance.”

Part of this process is just the normal course of events, whereby more powerful machines replace less capable ones and provide for increased performance. But this can create a false sense of complacency.

“When people centralized servers, the increased latency and protocol inefficiencies caused response times to tank,” says Joe Skorupa, research vice president of Enterprise Network Services and Infrastructure at Gartner. “With the introduction of WAN optimization tools, they had a two- or three-year honeymoon because response times improved so much that no one cared about measuring utilization or deploying quality-of-service metrics. But then the situation gets worse, and the latest rich Internet applications and peer networks drive up response times and compete with existing production applications.”

Sometimes, an offense is the best defense. EvriChart, a company that manages health information for hospitals, clinics and physician offices, hosts large quantities of scanned medical records over the Internet. Originally, it used proprietary software built around remote procedure calls that “were extremely inefficient,” says Tony Maro, the CIO of the company, which is based in White Sulphur Springs, W.Va. “After we redesigned the system to use HTTP and NFS, we continued to grow exponentially with no performance problems.”

Part of this redesign was a strategy that Maro used to put together a combination of open-source tools such as Nagios, Pound (a reverse proxy) and BandwidthD that help him monitor network latency and other statistics. “Constant monitoring has helped us solve issues before they became problems,” he says.

Another complicating issue is encryption. Many enterprises are making more use of Secure Sockets Layer (SSL) protocols to encrypt traffic as it traverses the Internet—both for their own protection from potential hackers and to maintain their customers’ privacy. But this presents challenges for applications infrastructure, particularly reverse proxy servers and load balancers, which may not be designed to handle encrypted traffic.

“As more of our traffic migrates from port 80 to port 443, we’ll have to decrypt the traffic so we can inspect it as it passes through the proxies,” says Wells Fargo’s Littlejohn.

Monitoring tools also should work with encrypted traffic. “You need to get some seriously enhanced visibility on ports 80 and 443,” says David O’Berry, director of ITSS at the South Carolina Department of Probation, Parole and Pardon Services in Columbia, S.C. “Next-generation firewalls from companies like Palo Alto and the devices from companies like Blue Coat are great first and second steps down that path. Combine that with additional application intelligence with incoming and outgoing traffic, and you can get the beginnings of a solid picture.”

Another issue is being able to scale up as more users run your applications and to understand what resources are needed to support them. Here’s how EvriChart’s Maro approached this situation: “If new applications are to be fully useful in the Web 2.0 world, they have to be designed to run in a truly distributed environment, and you have to make sure that it can quickly scale, too. We host 5 million pages of scanned documents.”

Part of Maro’s success has to do with how he built his servers out of common, inexpensive components. “We can throw another pair of mirrored servers in place and handle additional growth quite easily,” he explains. “Plus, given that our most expensive server is less than $2,500, the money we spend on our hardware is less than the maintenance fees on one of our competitor’s systems.”

A number of vendors sell tools to improve network applications throughput, including Blue Coat, Citrix, F5 Networks, Juniper Networks, Radware and Riverbed Technology. It can be hard to sort out which product will work the best for your firm.

“It’s more than just simple load balancing,” says Gartner’s Skorupa. “You can’t live without these products. In addition to significantly improving performance and reducing server loads, they are ideally positioned to look at coming and going traffic and find performance and security problems.”

Another complicating factor is the increasing role that virtualization plays in delivering enterprise applications. As more virtualized servers are turned on, the load balancing, proxy servers and other tools have to coordinate their movements and be able to work with this new infrastructure.

Wells Fargo’s Littlejohn talks about how “load balancing is really about finding the most effective way to deliver content, and its intent is to deliver a solution that virtualizes a service.”

The separate tools are now starting to combine into something more integrated. “Pure load balancing will evolve into a content delivery system, providing customers with predictable levels of performance,” she says. “In the WAN optimization space, a seamless integration is key. The manufacturers appear to be converging on a common feature set that includes transparency and effective replication of LAN services. Technologies will continue to evolve, and the transition to application delivery controllers will likely be part of the evolution.”

O’Berry of the Department of Probation adds: “Sometimes it seems more voodoo than science, although it continues to get better and better. But the days of the pure load-balancing appliance are numbered, and we are going to need something more integrated that goes up the entire applications stack.”

Improving applications can dramatically enhance performance. Take the case of Virgin America, an airline headquartered in San Francisco. Prior to upgrading its reservation system, on peak days, Virgin would reach 80 percent of its bandwidth capacity as customers bought tickets online. After deploying Radware’s AppDirector load balancer and upgrading its servers, Virgin’s pipes are now below 5 percent of their capacity, and the airline can focus on improving its customers’ Web experience.

“You have to balance saving money with making sure your business is supported effectively,” says Ravi Simhambhatla, director of IT infrastructure for Virgin America.