Making ConnectionsBy David F. Carr | Posted 2004-01-16 Email Print
Poor on-time performance was crippling America West. Then the airline dumped its failing hardware.
The SOC is one of two critical chokepoints that can either prevent delays (when functioning properly) or cause them (when it goes offline). The other is the hub operations room that still resides within the Phoenix airport. While the SOC's focus is nationwide, hub operations boil down to juggling gate assignments for incoming planes, getting departures off on time and trying to ensure that passengers make their connections. As America West's main hub, Phoenix's efficiency impacts most of the airline's passengers.
Hub operations stayed within the airport, but America West upgraded the local-area network and moved its servers to the data center. The other big step America West took with hub applications was to switch server vendors.
In 2000, America West implemented sophisticated new software to help hub operations staff optimize gate assignments. The gate-manager software from Ascent Technology of Cambridge, Mass., uses artifical-intelligence techniques to divine the best way to squeeze planes into the airport's limited real estate. For example, the software has to be able to account for the fact that the wingspan of larger jets means that two of them can't be parked next to each other at adjacent gates. It also helps hub operations personnel juggle gate assignments when planes arrive early or late.
This replaced a more chaotic system in which a room full of people would watch while one person standing on a platform moved around magnetic markers representing planes on a white board with a map of the airport.
However, gate management was one of several key hub operations applications on a Sun E10000 Starfire server that kept crashing. The problem was a manufacturing flaw related to the particular generation of Sun's UltraSparc processors used in the Starfire and other Sun computers, and the associated cache memory subsystem. "It was called the 'flipped bit' problem," Beery says, spitting out the words with distaste.
It happened when cosmic rayshigh-energy radiation from the heavenscalled alpha particles zapped electrons off course within the microprocessor. That caused electronic registers that represent bits of data or logic to "flip" 1 to 0 or 0 to 1and made the operating system "panic." About once a month, applications like gate management were knocked offline for an hour or more each time.
Sun acknowledged the problem in 2000. In response, it implemented "mirroring" of static random access memory (SRAM) chips so that, even if operating-system instructions are corrupted, the server can recover. Today, Sun believes the issue is irrelevant to its current offerings, having been solved more than two years ago. Sun spokeswoman Cinthia Portugal also emphasizes that "bit flips are not unique to Sun."
Still, the incident convinced America West to stop paying a premium for Sun servers. Over the past couple of years, the airline has migrated applications to Fujitsu's PrimePower servers, which run on Sparc-compatible processors and Sun's Solaris operating system.
America West purchased its first Fujitsu server in late 2001 and expanded the relationship to include system-management services for all Unix servers, including Sun products. When Fujitsu suggested its own hardware as an alternative to Sun's, Beery was ready to listen. He had joined America West as application development director in 1999 and had just been promoted to chief information officer in 2001. Even in his earlier development role, he had been annoyed by the server problems because "those were my applications that were crashing."
Beery says his team has learned to lower the cost of operating its information systems, even while improving reliability. He declined to release specific technology-budget numbers, but says the effect can be seen in the airline's overall performance. America West has whittled its cost per available seat mile (casm) from 8.20 cents at the end of 2002 to 7.57 cents in the three months ended Sept. 30. In that same quarter, Southwest and American Airlines reported casms of 7.51 cents and 9.43 cents, respectively.
Beery has no secret recipe, other than persistence and attention to detail. "It's about doing the right things with the tools you have, staying focused and working with some of the suppliers," he saysand about making sure it all adds up to better performance.