The Ugly History of Tool Development at the FAABy Edward Cone | Posted 2002-04-09 Email Print
Online exclusive: The agency wrote off $1.5 billion of its $2.6 billion investment to overhaul the nation's air traffic control computer systems. What went wrong? (Just about everything.)
One participant says, "It may have been the greatest failure in the history of organized work."
Certainly the Federal Aviation Administration's Advanced Automation System (AAS) project dwarfs even the largest corporate information technology fiascoes in terms of dollars wasted. Kmart's $130 million write-off last year on its supply chain systems is chump change compared to the AAS. The FAA ultimately declared that $1.5 billion worth of hardware and software out of the $2.6 billion spent was useless.
The AAS was supposed to provide a complete overhaul of the nation's major air traffic control computer systems, from new tools and displays for controllers to improved communication equipment and a revamped core computer network. It didn't even come close.
For years after the AAS was terminated in 1994, air traffic controllers were still using systems so old they ran on vacuum tubes. About $1.1 billion worth of the work that went into the AAS was salvaged through follow-up programs such as the Display System Replacement (DSR) project that provided new workstation designs to the En Route centers that deal with airplanes in the middle of their flights. But the AAS project as a whole failed. An aging mainframe system was left untouched, instead of being replaced.
What went wrong? Just about everything. Here's how the General Accounting Office put it: "FAA did not recognize the technical complexity of the effort, realistically estimate the resources required, adequately oversee its contractors' activities, or effectively control system requirements."
The project was probably doomed on the drawing board by an unrealistically ambitious plan. "It was basically a Big Bang approach, gigantic programs that would revolutionize overnight how FAA did its work," says Pete Marish, a senior analyst at GAO.
"We royally screwed up AAS, no doubt about it, in any way that a project could be screwed up," says FAA Associate Administrator for Acquisitions Steve Zaidman. "The benefits of AAS came at the very end," he says, and since the project never reached the intended finish line, most of them never materialized. In contrast, today the FAA is approaching the problem of systems development with a series of smaller projects that are supposed to deliver continual, incremental benefits.
Things did not start out well for AAS, which was conceived in 1981, at about the time that the Reagan administration broke a strike by air traffic controllers and fired more than 11,000 of them. The long procurement process (since changed to a more efficient acquisition method) led to a protest from a disappointed vendor, Hughes Electronics. That put the project a year behind before it really got going. "That was the year to develop the requirements," says Bill Krampf, director of engineering at Lockheed Martin, which eventually acquired IBM Federal Systems, a key AAS contractor. "We entered (the) software phase without the requirements phase completed."
In short, the project was probably too immense and unmanageable to begin with. The air traffic control system includes 173 terminal radar approach control centers (TRACONs) and 20 En Route centers, plus 460 federally managed control towers. It has to operate at high levels of performance and reliability, and there's almost never a chance to shut it down for maintenance. With the benefit of hindsight, most experts say the only way to overhaul this system is with a gradual, phased approachthe opposite of the tack taken with AAS.
The overambitious agenda was aggravated by excessive faith in new technologies. Among other things, AAS was supposed to be a showcase for Unix-based distributed computing and for development in Ada, a programming language created by the Air Force that became the state-sponsored religion in object-oriented technology, itself a relatively young methodology for writing code in self-contained, reusable chunks. IBM tried to use Ada to enforce discipline on the project by making developers outline a design in high-level code, then fill in the blanks. But this was no match for an environment of where the FAA kept changing its requirements.
Robert Britcher, now a systems engineering professor at Johns Hopkins University who worked on AAS for IBM and wrote about it in his book, "The Limits of Software: People, Projects, and Perspective," from which comes the quotation about the project's place in the history of work. He says that in contrast to the FAA's first, punch-card-driven air traffic control computer systemswhere the difficulty of the job was clearly recognizedIBM and the FAA approached AAS as if it ought to be relatively easy, thanks to object-oriented programming languages, modern development tools and distributed systems.
Now that computer programs had escaped from laboratories and mainframe computer rooms, expectations for software had become tremendous, Britcher says. "Everybody expected that it could be something that it's notlike flexible and easy to manage and implement."
IBM's confidence also encouraged the FAA to set what turned out to be unrealistic requirements, he says. "For example, they wanted the system to have only 3 seconds of downtime a year," Britcher says. "But to get the data to prove that requirement had been met would have taken about 10 years."