Implementing VOIP on a Crazy Schedule: A Mini Case Study

By David F. Carr  |  Posted 2007-09-11

For Molina Healthcare, the cost savings associated with Voice Over Internet Protocol technology proved less important than the flexibility it offered—flexibility for disaster response, as well as more routine administration and reconfiguration of the phone network.

Molina's VOIP project succeeded despite a breakneck schedule, malfunctioning equipment, and the sale of the consulting firm hired to perform the work. And before the project was over, the technology would prove its worth not just as a way to cut telecommunications costs but for the flexibility it offered in the face of what could have been a day-long business interruption.

The Long Beach, California-based healthcare company has been growing rapidly in recent years, with about 85 percent of that growth coming from acquisitions. As it acquired other health plans, Molina, which specializes in serving low-income people, also inherited a hodgepodge of telecommunications systems and carrier contracts, leading to higher toll charge and administrative costs. By switching to a Cisco VOIP system, in which corporate phone traffic would flow over the same network as data, Molina aimed to standardize operations, cut costs, and increase flexibility.

Like many organizations with operations in multiple states, Molina was attracted to VOIP partly because of the potential to avoid long-distance charges by routing phone calls over its corporate wide-area network, rather than the public telephone network. But VOIP systems are also more programmable and easier to reconfigure.

For example, the relationship between an extension number and a phone sitting on a particular desk is set in software, rather than hard-wired in physical circuits. So employees can sit down at a different desk and "log in" to the phone there, and calls to their extension will immediately start ringing through to that phone. For their part, administrators can make sweeping changes through a web-based administrative tool.

The combination of long-distance savings and simplified administration can help drive down costs. How much savings is a question, however. Jay Lassman, an analyst with Gartner Inc., says VOIP vendors often play a "shell game" by shifting telecom expenses into the data communications budget as a way of making the savings look larger.

In Lassman's view, the real benefits only materialize if VOIP is pursued in the context of a broader initiative to improve communication and collaboration. "If you're just doing a like-to-like replacement of an older phone system with VOIP, you're only looking at maybe a 3% ROI," Lassman says.

The VOIP project grew out of a 2003 initiative to upgrade the phone systems at Molina's headquarters, which in turn led to an examination of how a company-wide upgrade to a common digital phone system could improve Molina's efficiency and its disaster recovery capabilities. Molina currently operates in eight states.

The VOIP implementation began in April 2005 and ended in November of the same year—a very tight schedule. "We had people sleeping in conference rooms after pulling all-nighters" on a few occasions at sites where the implementation was going badly, says Edward Poman, Molina's manager of network infrastructure.

The tight schedule so worried former Chief Information Officer Rick Click, who joined the company in May 2005, about a month into the implementation, that he considered canceling it, even though the contracts were already signed. (Click left Molina last month.)

One of the things that bothered Click was the plan to complete the rollout within six months, which struck him as overly ambitious. "It easily should have been a 12- to 18-month project," to allow time to adjust plans if things went wrong along the way, he said in an interview earlier this year. He said the schedule was set by another (also now former) executive of the company who was determined to show that the information technology department could move more quickly and deliver projects on time and under budget.

Meanwhile, the consulting firm Molina had hired for the project, Norstan of Minnetonka, Minn., had changed hands. In early 2005, shortly after being acquired by Black Box Network Services of Lawrence, Penn., Norstan's Cisco Consulting practices was sold to Spanlink Communications Inc. of Minneapolis, Minn.

Click said he was convinced Norstan underbid the project and over-committed on promising how quickly it could be completed, but nevertheless Spanlink promised to follow through on those commitments. Still, the burden wouldn't be just on Spanlink but also on Molina personnel who would have to work long hours, on an unforgiving schedule, to pull the project off. "We really cautioned a slowdown," said Jeremy Chapman, Spanlink's Technical Services Director.

Early Problems

The first implementation, at the call centers for the state of Washington in Bothell and Spokane, was one of the harder ones. As call center agents began to use the technology, they complained about intermittent problems with voice quality that made it hard for them to do their jobs.

That's one of the core challenges of VOIP, since the sound of every word spoken must be broken into a digital representation, with those digits assembled into Internet protocol packets and streamed across a network — all of which is much different than delivering voice over continuous circuits in a traditional phone system.

If voice data packets are delivered at an irregular pace, voices may be distorted and can wind up being unintelligible. So it's important to make sure the underlying network supplies both the necessary bandwidth and mechanisms for ensuring voice traffic gets the priority treatment it requires.

The project leaders thought they had done that. It took some sleuthing to uncover the problem.

In conjunction with the VOIP rollout, Molina was upgrading its wide-area network to MultiProtocol Label Switching (MPLS) from frame relay, an older data communications standard. The new AT&T MPLS service provided the means to differentiate between the quality of service required by data traffic, which can wait, and voice traffic, which can't. In the process, bandwidth to some locations was also increased by as much as 50 percent.

One thing that made the voice problem hard to figure out was that it was intermittent, seeming to go away after the team fiddled with the network or provided the call center agent who reported the problem with a new phone, and then cropping up again.

The answer turned out to be something outside the network—in fact, it wasn't a VOIP problem at all. It turned out that more than one in eight of the headsets provided to the call center agents was defective—rather than any network connection failing, the problem was somewhere in the few feet of cable and wiring between the Cisco phone on the agent's desk and the speaker of the headset earpiece.

Molina declined to name the headset vendor, except to say the firm was on an approved vendor list from Cisco, and Cisco provided one of its top troubleshooters at no charge to help unravel the problem.

As the project rolled forward, things started to go more smoothly.

Larry Santucci, director of enterprise services at Molina, started getting equipment delivered from the manufacturer already partly configured, and doing most of the remainder of the configuration at headquarters, prior to visiting the site of each installation.

He and Poman, along with Spanlink personnel, also learned to stress-test the VOIP systems on a Thursday or Friday night before the system was scheduled to go live, leaving themselves the weekend to resolve any problems. Then, after each deployment, a small post-implementation team would be left behind at each location to do training and troubleshooting.

So, no more all-nighters, just a concerted 9 to 5 push to keep the project moving. Then, as the project was nearing its end in October of 2005, one incident showed demonstrated the value of the VOIP system.

On Oct. 17, 2005, Molina lost local phone service to its Long Beach, Calif., headquarters that affected four call centers serving California members and several ancillary services, such as a nationwide nurse help line members can call to get answers to common medical questions. The outage was caused by a problem with switching equipment operated by Verizon, the local phone carrier for the area, and it lasted for an entire day. In the past, this would have resulted in a major business disruption.

But with the VOIP system now in place throughout most of Molina's corporate network, the mere fact that there was no local phone service to the Long Beach facilities didn't have to mean that no calls could get through.

The solution was to get Molina's long-distance carrier, AT&T, to reroute calls coming in to the company's toll-free customer service numbers for California members, sending them to a Molina facility in Albuquerque, New Mexico. There, the calls were translated into VOIP traffic and rerouted back to Long Beach. Thanks to the Cisco Call Manager system, rerouting the calls was as easy as making a few changes through a web-based administration screen.

Nothing of the sort would have been possible without VOIP, Poman says. At best, in prior years, calls might have been re-routed to a facility such as the New Mexico one and handled there. And even that would have been awkward. "If each location has different telephone systems in place, it becomes a real challenge to do it for a day," Poman says.

Also, in that scenario, California customers would have been left wondering why their calls were being taken by New Mexico customer service representatives, and they probably would have been confused by differing phone tree menus and recorded messages. With one common VOIP system, Molina has been able to make all those things more consistent, so that different locations are able to serve as backup locations for each other, Poman says.

In fact, several times this winter when facilities were forced to close because of snowstorms, Poman was able to reroute traffic from the affected facility to another that remained open. And by doing it through the corporate VOIP system, rather than asking a phone carrier to reroute calls, he saved both time and money. In other words, the whole process became a configuration change his staff could make from within a web-based administration tool, rather than depending on the phone company to do it for them.

"With VOIP, we handled everything internally," Poman says.

Payoff and New Possiblities

Ultimately, the VOIP project played well within a broader corporate strategy of consolidating core systems to lower operating costs. Particularly in a company growing through acquisitions, and trying to improve the efficiency of each newly acquired health plan, targeting the "general and administrative" portion of the corporate financial statement is key, Poman says.

The VOIP project helped drive down costs in many ways. For starters, consolidating the multiple telecommunications contracts previously negotiated by separate plans shaved about 13 percent off the total, Click says. Molina says the project has now paid for itself with savings over and above the cost of implementing the VOIP system.

While giving a tour of the Long Beach data center some months ago, Poman and Santucci pointed out the amount of floor space they've recovered, with modern equipment that provides more functionality and fits in less than half the space that their old Private Branch Exchange system occupied.

Meanwhile, the flexibility afforded by VOIP technology is proving useful both for disaster recovery (including minor "disasters" like snowstorms and phone outages) and for routine operations. For example, the nurses manning Molina's nurse help line can now work from home, using a Cisco phone and a broadband Internet connection to tie into the corporate phone system. That in turn is making it easier for Molina to recruit nurses with specific skills, such as fluency in Spanish, because it minimizes the importance of geography.

Similarly, the information technology department is considering virtualizing its computer help desk by tapping personnel stationed in other offices to help field calls coming in to the main help desk in Long Beach.

Also, now that Molina has gained experience re-routing customer service calls between offices during events such as snow storms, it is starting to consider doing so on a more routine basis—simply to shift work between one office that is overloaded to another that is less busy. That would require more cross-training between staffs—so that New Mexico personnel really would know how to field calls from California customers, for example—but the technology is in place to make it happen.