The Ugly History of Tool Development at the FAA

 
 
By Edward Cone  |  Posted 2002-04-09
 
 
 

One participant says, "It may have been the greatest failure in the history of organized work."

Certainly the Federal Aviation Administration's Advanced Automation System (AAS) project dwarfs even the largest corporate information technology fiascoes in terms of dollars wasted. Kmart's $130 million write-off last year on its supply chain systems is chump change compared to the AAS. The FAA ultimately declared that $1.5 billion worth of hardware and software out of the $2.6 billion spent was useless.

The AAS was supposed to provide a complete overhaul of the nation's major air traffic control computer systems, from new tools and displays for controllers to improved communication equipment and a revamped core computer network. It didn't even come close.

For years after the AAS was terminated in 1994, air traffic controllers were still using systems so old they ran on vacuum tubes. About $1.1 billion worth of the work that went into the AAS was salvaged through follow-up programs such as the Display System Replacement (DSR) project that provided new workstation designs to the En Route centers that deal with airplanes in the middle of their flights. But the AAS project as a whole failed. An aging mainframe system was left untouched, instead of being replaced.

What went wrong? Just about everything. Here's how the General Accounting Office put it: "FAA did not recognize the technical complexity of the effort, realistically estimate the resources required, adequately oversee its contractors' activities, or effectively control system requirements."

The project was probably doomed on the drawing board by an unrealistically ambitious plan. "It was basically a Big Bang approach, gigantic programs that would revolutionize overnight how FAA did its work," says Pete Marish, a senior analyst at GAO.

"We royally screwed up AAS, no doubt about it, in any way that a project could be screwed up," says FAA Associate Administrator for Acquisitions Steve Zaidman. "The benefits of AAS came at the very end," he says, and since the project never reached the intended finish line, most of them never materialized. In contrast, today the FAA is approaching the problem of systems development with a series of smaller projects that are supposed to deliver continual, incremental benefits.

Things did not start out well for AAS, which was conceived in 1981, at about the time that the Reagan administration broke a strike by air traffic controllers and fired more than 11,000 of them. The long procurement process (since changed to a more efficient acquisition method) led to a protest from a disappointed vendor, Hughes Electronics. That put the project a year behind before it really got going. "That was the year to develop the requirements," says Bill Krampf, director of engineering at Lockheed Martin, which eventually acquired IBM Federal Systems, a key AAS contractor. "We entered (the) software phase without the requirements phase completed."

In short, the project was probably too immense and unmanageable to begin with. The air traffic control system includes 173 terminal radar approach control centers (TRACONs) and 20 En Route centers, plus 460 federally managed control towers. It has to operate at high levels of performance and reliability, and there's almost never a chance to shut it down for maintenance. With the benefit of hindsight, most experts say the only way to overhaul this system is with a gradual, phased approach—the opposite of the tack taken with AAS.

The overambitious agenda was aggravated by excessive faith in new technologies. Among other things, AAS was supposed to be a showcase for Unix-based distributed computing and for development in Ada, a programming language created by the Air Force that became the state-sponsored religion in object-oriented technology, itself a relatively young methodology for writing code in self-contained, reusable chunks. IBM tried to use Ada to enforce discipline on the project by making developers outline a design in high-level code, then fill in the blanks. But this was no match for an environment of where the FAA kept changing its requirements.

Robert Britcher, now a systems engineering professor at Johns Hopkins University who worked on AAS for IBM and wrote about it in his book, "The Limits of Software: People, Projects, and Perspective," from which comes the quotation about the project's place in the history of work. He says that in contrast to the FAA's first, punch-card-driven air traffic control computer systems—where the difficulty of the job was clearly recognized—IBM and the FAA approached AAS as if it ought to be relatively easy, thanks to object-oriented programming languages, modern development tools and distributed systems.

Now that computer programs had escaped from laboratories and mainframe computer rooms, expectations for software had become tremendous, Britcher says. "Everybody expected that it could be something that it's not—like flexible and easy to manage and implement."

IBM's confidence also encouraged the FAA to set what turned out to be unrealistic requirements, he says. "For example, they wanted the system to have only 3 seconds of downtime a year," Britcher says. "But to get the data to prove that requirement had been met would have taken about 10 years."

Out of Control

Rather than admit the problem, IBM turned AAS into a research project on keeping distributed computer systems up and running on an almost nonstop basis, publishing papers instead of making concrete progress.

Deadlines vanished, and costs spiraled out of control. Only after the AAS project collapsed did the FAA set a more realistic requirement of no more than 5 minutes of downtime a year.

Another ex-IBMer who worked on AAS, Ellen Bass, says she quickly saw problems. "The science wasn't there, and the methodologies weren't there," says Bass, who spent the years following AAS' demise working as a human-factors specialist on submarine and aircraft control systems and now teaches at the University of Virginia. In addition to the unsolved problems of around-the-clock distributed computing systems, the AAS designers were venturing too far into uncharted territory with her specialty, the user interface.

Bass saw that a then cutting-edge plan to allow controllers to customize their workstations would make it virtually impossible to test whether the system would operate safely in every configuration. For example, by giving controllers the opportunity to customize the colors on their display, the design introduced the possibility that they might accidentally make planes appear invisible against the background. She suggested going back to the FAA with a proposal for a "simpler, cheaper, quicker" solution, but she was a junior member of the team whose opinions didn't count for much. Besides, her managers at IBM were "very concerned that we had promised FAA all this flexibility, and if we simplified we would be non-compliant," she says.

There were internal FAA politics at work, too. After the demise of PATCO (the Professional Air Traffic Controllers Organization), the FAA looked to computer automation as a means of allowing fewer controllers to handle more planes, and that was part of what AAS was supposed to accomplish. But before long there was a new union, NATCA (the National Air Traffic Controllers Association), which had every reason to frustrate that goal.

On paper, the AAS was also supposed to result in the consolidation of what today remain two separate functions performed by different facilities, the TRACONS that handle airport arrivals and departures and the En Route Centers that handle the long-haul portion of a flight.

The FAA wrote the consolidation plan into the project specifications without solving the political problems that would have to be overcome to make it happen. So the consolidation plan was eventually dropped, but the requirement that IBM produce a new workstation capable of handling both sorts of functions remained in the specs. "We were working toward a mission that was not even there," Britcher says. In other words, developers were creating a unified workstation for two functions that were now no longer going to be unified.

FAA officials say they have learned from the failure of the AAS and will not repeat the same mistakes. But one of the project's legacies is an enduring cynicism, from many fronts, about anything that FAA officials say.

"The FAA says it sees light at the end of the tunnel, but when you get to the end of that tunnel, there's another tunnel," says David Schaffer, the counsel to the U.S. House Aviation Subcommittee. Whenever he hears claims of progress, he can't help but think of what he kept hearing about the AAS until the very end.

"They would say there were a few problems, but they were being worked out. Everything seems to be going well—until it collapses."