What Went WrongBy John McCormick | Posted 2004-03-04 Email Print
Know the Risk: Digital Transformation's Impact on Your Business-Critical Applications REGISTER >
Additional reporting by Berta Ramona Thayer in Panama
As software spreads from computers to the engines of automobiles to robots in factories to X-ray machines in hospitals, defects are no longer a problem to be managed. They have to be pred
Overlooking the skyscrapers of downtown Panama City, amid towering palm trees and gracious homes in the old Canal Zone, sits Gorgas Hospital, an imposing concrete structure which now houses Panama's National Cancer Institute. This is a public hospital. No Panamanian is turned away. On a Monday morning in January, at least 50 patients and their family members, including Victor Garcia and his wife, are visiting the institute. The patients walk slowly up the driveway; sit quietly on the patio under the lush vegetation that surrounds the building; stand in the lobby. They are all waiting for treatment.
This is not even the hospital's busiest day of the week. That is Tuesday, when the clinic offers every citizen, even those without a doctor's referral, free diagnoses of the skin cancer that tends to flourish under the equatorial sun.
Cancer is a leading cause of death in Panama: prostate cancer for men, endometrial and cervical cancer for women. And those are unlikely to be just the sun's fault. Many Panamanians blame the United States' testing of the chemical defoliant Agent Orange in the Canal Zone during the Vietnam War. Since 1997 the number of new cancer patients in Panama has more than quadrupled, according to the cancer institute. The hospital now sees 10 to 15 new patients per day and performs 300 cancer surgeries per month.
The victims of the faulty radiation treatments in 2000 and 2001 span the breadth of Panamanian society. Among the dead are Margarita Sevillano, a folksinger; Walter Chandler, a professor at the University of Panama; and Rosa Vergara, a nun. Many of the dead lived in the barrios in the hills above downtown, where chickens peck along the roads, laundry flaps from porches and brightly painted stucco houses are interspersed with small shops and Internet cafés.
The hospital's radiotherapy unit is critical to Panama. When the IAEA's investigation, in May 2001, slowed the hospital's routine, patients lined up waiting to be treated. That led the Panamanian ambassador in Austria, Jorge Perez, to urge the Vienna-based agency to hurry up. "Those who could afford to went to the private clinics," Perez says. "Those who could not, waited."
The difference in cost to the Panamanian government is stark. Garcia's treatment, for example, which cost him virtually nothing at the National Cancer Institute, would cost $4,000 at a private hospital, using a Cobalt-60 machine. Using a higher-powered, more-precise linear accelerator, the bill would escalate to $10,000.
The current chief of the cancer institute's radiotherapy unit, Dr. España de la Rosa, asserts that some patients died in 2001 waiting for treatment, while the overdoses were being investigated. She says she does not know how many.
But the survivors of the overdoses didn't fare well, either. The governments of France and Argentina each offered to take two of the over-radiated patients and treat them for a year at no charge. Panama sent no one. "We are a small country, and everybody knows everybody," Ambassador Perez says. "How do you decide who to send?"
The overdoses occurred not in the newly renovated Gorgas Hospital on the hill, but in the cramped Justo Arosemena Avenue facility downtown, which the hospital was in the process of vacating. The Multidata software and the Cobalt-60 teletherapy machine manufactured by Theratronics had been installed there in 1993. According to a letter written to Multidata by ProMed, the Panamanian distributor that sold the hardware and software, the hospital was looking for cheaper software because it couldn't afford the software that Theratronics typically supplied with its radiation machine.
ProMed services manager Camilo Jorge says he doesn't remember the price difference, but he knows the hospital never purchased a maintenance contract for the software-only for the radiation machine. By 1997, hospital staff was so concerned about the possibility of unintended excess exposure that they warned in a report requested by the Ministry of Health of "overexposure of radiation-therapy patients due to human error" unless conditions at the hospital improved.
In the report, the staff claimed the hospital was understaffed and poorly equipped, and they asked for more frequent maintenance on the Cobalt-60 teletherapy machine. The contention: the machine was being used 3,780 hours per year, nearly twice what the maintenance program recommended.
The staff also asked the hospital to have Multidata do "preventive maintenance" on the software. But the software was never maintained, and by 2000, the hospital was using just the one Cobalt-60 machine to treat all patients, according to Saldaña. A second, older machine was retired.
By then, two of the hospital's five radiation physicists had quit. Saldaña says the remaining three did the work of five, which sometimes required 16-hour days.
Victor Garcia remembers waiting five to six hours for every treatment. And after each of those six treatments, he felt sicker. As his intestines struggled to slough off cells killed by the radiation, he developed diarrhea. Burns seared through the flesh on his back. He lost 30 pounds. Hospital doctors told him the symptoms were normal. One reason it took hospital staff seven months to discover the overdoses, according to hospital director Juan Pablo Bares, is that patients with pelvic cancers often show symptoms of radiation toxicity, and the number of patients overdosed was small compared to the number being treated.
But another reason was the complexity of the software. The glitch involving Multidata was activated only under very specific circumstances-when the dimensions of the blocks that defined the patient's treatment area were entered in a particular way. If the blocks were treated as a single, composite shape, and the descriptions of their dimensions were entered so that the "loops" that defined the inner and outer perimeters of that shape crossed, the software would increase patients' treatment time, the IAEA report said.
As patients began to sicken and then die, the staff hunted for the cause. Saldaña remembers that by March 2001, she was thinking the problem had to be the software. But even then she discovered it by accident: On the morning of March 2, according to a statement she gave to the prosecutor's office, she was calculating dosages for two patients with equivalent treatment areas and treatment depths and suddenly realized that the treatment times that came out of the software weren't even close.
And so began the hospital's effort to unearth the causes of the overdoses.
Radiotherapy expert J. Francisco Aguirre, who investigated the overdoses for the Panamanian government as part of a team from the M.D. Anderson Cancer Center in Houston, says the calculation error was a problem that occurred with algorithms in older software used to plan treatments, a linkage that Multidata president Arne Roestel denies. Aguirre says the error was so obscure he wouldn't have thought to look for it-except that while he was in Panama, he remembered seeing a physicist in the U.S. cause a similar error 10 years before.
"The trick is how to tell the computer what are [empty] holes and what is solid," Aguirre says. "If the lines you are digitizing cross along the way, you fool the computer."
Indeed, during the IAEA's May 2001 investigation, the agency found ways to get the software to miscalculate treatment times that the hospital staff hadn't tried. Investigators were able to enter the dimensions for one block, two blocks or four blocks of varying shapes, and every time they treated them as a single block and entered the coordinates so that the perimeter loops crossed, the software always increased the treatment times.
Both the M.D. Anderson and IAEA investigating teams found Multidata's manual hard to understand. "It does not describe precisely how to digitize co-ordinates of shielding blocks and there are not enough relevant illustrations," the IAEA report said. "In addition, it does not provide specific warning against data entry approaches that are different from the one described."
The Houston team's report said: "The manufacturer's manual of instructions was reviewed, and no indication was found in the instructions on how to digitize the blocks, or procedures to avoid, that could result in bad calculations."
On Aug. 10, 2001, in an "urgent notice" to users, Multidata used a series of diagrams to describe how the "crossing-loop" problem-which the company described as a "data entry sequence that creates a self-intersecting shape outline"-would not be acceptable to its program and would cause miscalculations. And it appeared to specifically absolve those users who, like Saldaña, had tried to get the software to give results when five shields were being placed on patients instead of four.
"Digitizing direction and exceeding the number of blocks, numbers of points per block or the block shape have no unexpected effect on the dose calculation," the notice said.