Fidelity Investments, the Boston-based financial services firm, uses voice-over-IP technology in most of its international offices to save on long-distance charges to other Fidelity locations. To date, it’s put IP-based phones on the desks of about 10% of its 30,000 employees.
Mike Brady, Fidelity’s senior vice president of telecommunications, leads a team that runs the voice networks and ensures calls meets the company’s quality standards. It’s a tough job, not least of which is because he’s usually thousands of miles away from the people who need his help.
But the trickiest part, Brady says, stems from the fact that data networks were not intended to carry voice. They’re resilient for data: IP networks automatically reroute transmissions if one network segment fails. The trouble is, that trait can absolutely murder phone calls, because increasing the time it takes for the data that comprises a voice call to reach its destination—by even a quarter of a second—can turn a conversation into gibberish.
Traditional data network management tools can’t always provide enough information to see glitches that affect voice. Using regular network monitoring systems, “we can understand delays between any two points in a network,” Brady says. “But with voice-over-IP, that’s no longer good enough.”
Fidelity uses a technology called quality of service, which instructs a network to always send voice traffic ahead of other data, to minimize the chance of delays. But today such quality-of-service information doesn’t carry over from one network to another (although newer technologies, such as Multiprotocol Label Switching, or MPLS, promise to do this once they’re widely deployed).
Brady had to find a better way to crack this nut. So, for the past year, Fidelity has used monitoring equipment from Brix Networks, a small Boston-area firm, to catch degradation of IP phone service before complaints start trickling in. Brix’ boxes place simulated voice calls every few minutes from various points on the network, and can detect when sound quality is going south. “Brix helps us find and fix the ‘gray’ failures,” Brady says.
A cottage industry has sprung up around ensuring quality for voice-over-IP networks. Specialists attacking this problem include startups such as Brix, Qovia and Viola Networks; data network monitoring tools providers like Apparent Networks, Integrated Research and NetIQ also provide products for monitoring IP telephony. In addition, major vendors of IP phone equipment, including Avaya and Cisco Systems, sell voice-quality assurance tools along with their systems.
Such products are crucial for many organizations that have moved to IP telephony and that used to rely on third-party technicians, or a phone company, to manage their voice service.
“We are, in essence, our own telco now,” says Norm Baxter, an information-technology project leader in the city of Mississauga in Ontario, Canada. The city two years ago replaced phone lines supplied by Bell Canada with a Cisco phone system. On the plus side, he notes: “We don’t have to wait 48 hours for a response from Bell Canada to find out what’s going on with the system.”
Problems will, inevitably, arise. Normally, quality of service policies ensure that voice traffic gets through, regardless of what’s happening on the rest of the network. But even with those best-laid plans, an IP telephony system can be affected by network anomalies, such as a malfunctioning network card or a virus that causes a PC to flood the network. “Some things just strain the network too much,” says Matt Brunk, president of Telecomworx, a voice network consulting firm in Monrovia, Md., that uses Qovia’s monitoring tools.
For Tennille Spence, information-technology business liaison with the General Services Administration regional office in Kansas City, Mo., managing the division’s Cisco IP telephony system used to be a big hassle: She’d sometimes spend up to 16 hours per week hunting down the causes of problems with the phone system, involving service provider connections at the GSA’s field office across town.
This summer, Spence deployed probes from Qovia that can place test calls and instantly analyze where there may be bottlenecks or errors along the path. She’s also set up Qovia to send alerts to her BlackBerry mobile e-mail device if anything goes haywire after hours or on the weekend. “This really frees up my time,” she says.
Of course, sometimes there’s just no pleasing people. Many employees have a preconceived notion that any IP-based phone call will sound terrible, says Michael Burrell, senior manager for enterprise telephony at Equant, a provider of voice and data network services. He recalls telling an executive in the company’s Paris office two years ago that her phone had switched to a voice-over-IP system. She refused to believe him—because she hadn’t noticed a change in voice quality.
“A lot of it is perception,” Burrell says. “If you tell people you’re cutting over to IP telephony over the weekend, when you come in Monday morning you’re going to have a lot more trouble tickets than if you hadn’t announced it.”
