Early ProblemsBy David F. Carr | Posted 2007-09-11 Email Print
WEBINAR: On-demand webcast
Next-Generation Applications Require the Power and Performance of Next-Generation Workstations REGISTER >
The healthcare provider Molina gave itself six months to replace a tangle of inherited telecommunications systems with a single one built on IP. Here's how it managed the feat.
The first implementation, at the call centers for the state of Washington in Bothell and Spokane, was one of the harder ones. As call center agents began to use the technology, they complained about intermittent problems with voice quality that made it hard for them to do their jobs.
That's one of the core challenges of VOIP, since the sound of every word spoken must be broken into a digital representation, with those digits assembled into Internet protocol packets and streamed across a network — all of which is much different than delivering voice over continuous circuits in a traditional phone system.
If voice data packets are delivered at an irregular pace, voices may be distorted and can wind up being unintelligible. So it's important to make sure the underlying network supplies both the necessary bandwidth and mechanisms for ensuring voice traffic gets the priority treatment it requires.
The project leaders thought they had done that. It took some sleuthing to uncover the problem.
In conjunction with the VOIP rollout, Molina was upgrading its wide-area network to MultiProtocol Label Switching (MPLS) from frame relay, an older data communications standard. The new AT&T MPLS service provided the means to differentiate between the quality of service required by data traffic, which can wait, and voice traffic, which can't. In the process, bandwidth to some locations was also increased by as much as 50 percent.
One thing that made the voice problem hard to figure out was that it was intermittent, seeming to go away after the team fiddled with the network or provided the call center agent who reported the problem with a new phone, and then cropping up again.
The answer turned out to be something outside the network—in fact, it wasn't a VOIP problem at all. It turned out that more than one in eight of the headsets provided to the call center agents was defective—rather than any network connection failing, the problem was somewhere in the few feet of cable and wiring between the Cisco phone on the agent's desk and the speaker of the headset earpiece.
Molina declined to name the headset vendor, except to say the firm was on an approved vendor list from Cisco, and Cisco provided one of its top troubleshooters at no charge to help unravel the problem.
As the project rolled forward, things started to go more smoothly.
Larry Santucci, director of enterprise services at Molina, started getting equipment delivered from the manufacturer already partly configured, and doing most of the remainder of the configuration at headquarters, prior to visiting the site of each installation.
He and Poman, along with Spanlink personnel, also learned to stress-test the VOIP systems on a Thursday or Friday night before the system was scheduled to go live, leaving themselves the weekend to resolve any problems. Then, after each deployment, a small post-implementation team would be left behind at each location to do training and troubleshooting.
So, no more all-nighters, just a concerted 9 to 5 push to keep the project moving. Then, as the project was nearing its end in October of 2005, one incident showed demonstrated the value of the VOIP system.
On Oct. 17, 2005, Molina lost local phone service to its Long Beach, Calif., headquarters that affected four call centers serving California members and several ancillary services, such as a nationwide nurse help line members can call to get answers to common medical questions. The outage was caused by a problem with switching equipment operated by Verizon, the local phone carrier for the area, and it lasted for an entire day. In the past, this would have resulted in a major business disruption.
But with the VOIP system now in place throughout most of Molina's corporate network, the mere fact that there was no local phone service to the Long Beach facilities didn't have to mean that no calls could get through.
The solution was to get Molina's long-distance carrier, AT&T, to reroute calls coming in to the company's toll-free customer service numbers for California members, sending them to a Molina facility in Albuquerque, New Mexico. There, the calls were translated into VOIP traffic and rerouted back to Long Beach. Thanks to the Cisco Call Manager system, rerouting the calls was as easy as making a few changes through a web-based administration screen.
Nothing of the sort would have been possible without VOIP, Poman says. At best, in prior years, calls might have been re-routed to a facility such as the New Mexico one and handled there. And even that would have been awkward. "If each location has different telephone systems in place, it becomes a real challenge to do it for a day," Poman says.
Also, in that scenario, California customers would have been left wondering why their calls were being taken by New Mexico customer service representatives, and they probably would have been confused by differing phone tree menus and recorded messages. With one common VOIP system, Molina has been able to make all those things more consistent, so that different locations are able to serve as backup locations for each other, Poman says.
In fact, several times this winter when facilities were forced to close because of snowstorms, Poman was able to reroute traffic from the affected facility to another that remained open. And by doing it through the corporate VOIP system, rather than asking a phone carrier to reroute calls, he saved both time and money. In other words, the whole process became a configuration change his staff could make from within a web-based administration tool, rather than depending on the phone company to do it for them.
"With VOIP, we handled everything internally," Poman says.