The InvestigationsBy John McCormick | Posted 2004-03-04 Email Print
Re-Thinking HR: What Every CIO Needs to Know About Tomorrow's Workforce
Additional reporting by Berta Ramona Thayer in Panama
As software spreads from computers to the engines of automobiles to robots in factories to X-ray machines in hospitals, defects are no longer a problem to be managed. They have to be pred
Multidata's Mick Conley, in an interview with Baseline on Feb. 19, maintained that his company first heard full details of the overdoses from the FDA and the NRC in June 2001.
But Aguirre says that is not true. His team notified Multidata in April, he says, two days after they returned from Panama to the U.S.
"We told them, 'Your equipment has had an accident, go and make sure that for all the systems you've sold in the world, people are aware this is a problem,'" Aguirre says.
The IAEA report confirms that the Houston team notified Multidata in April, "and that it was impressed upon Multidata to send someone to Panama as soon as possible to resolve this problem." But Roestel says Multidata could not get enough information from the hospital or the Houston team on exactly what had happened.
In addition, Conley says, in order for the company to send representatives to Panama, the hospital would have needed a service contract with the company, which it didn't have, although he adds that Multidata provides telephone support to anybody with its product.
But before sending a team out, Conley says, "you really have to have an idea of what the problem is." If another customer called in a similar situation, he says, Multidata would want a clear explanation of what the problem was so that they "knew what kind of person to send to a place and what kind of tools to send along with them."
On May 18, after several patients had already died, the Panamanian government announced the overdoses, and international agencies began to act. In the U.S., the NRC sent out warnings on June 1 and again on June 6 alerting hospitals licensed to perform radiation therapy of the overdoses-and that Multidata's software was involved.
Multidata issued its first warning to customers on June 22, although it did not say which versions of its software were affected or how the overdoses happened. The company directed users to "follow instructions in the user manual...follow a written quality assurance procedure...and perform verification measures."
Multidata issued what Conley calls a "voluntary in-field correction," starting in the fall of 2001. The software patch checked treatment-planning calculations and rejected anything that was not identified by the system as a valid shape. "We changed the software so that [the Panama incident] could not happen again," he says.
Conley maintains the company was initially unaware the software was still being used in Panama. "We never heard from [the hospital]," he says. Conley and Roestel also contend the software was fine and that the problem was user error.
The IAEA report did note that quick action by the hospital staff could have prevented the overdoses.
At the end of May 2001, right after the FDA became aware of the accidents in Panama, it sent its examiners to inspect Multidata. The FDA found that Multidata had received at least six complaints about "calculation errors related to the failure of the firm's radiation treatment planning software to correctly handle certain types of blocks (polygons)."
The report said: "As of 6/1/01, there was no documentation that any complaint or incident report analysis had been performed, or corrective action developed or implemented [by Multidata]. In addition, the firm had been aware of this failure since at least 9/92."
Separately, the NRC published the findings of the IAEA in an Information Notice to operators of radiotherapy machines dated Nov. 20, 2001. The notice said: "Specifically, the staff modified its procedures and entered data for multiple shielding blocks together ('digitized' the blocks), as if they were a single block. The data were accepted by the treatment planning system, but the [Multidata] software calculated incorrect treatment times. Using incorrect treatment times resulted in significant radiation overexposure to patients."
In 2003, Multidata signed a consent decree with the FDA that precludes the company from making or selling software for radiation-therapy devices in the U.S., although it can still export its products. According to the FDA's injunction announcement, Multidata failed to meet the FDA's manufacturing practices and design standards.
Of all the criticism the firm received, perhaps the harshest was a statement made by FDA Commissioner Mark McClellan when the injunction was made public on May 7, 2003.
"Multidata Systems has a nine-year history of violations and failure to correct them," he said. "Despite repeated warnings, the company continued to manufacture its medical devices in a way which put the public health at risk."
Indeed, the FDA has taken Multidata to task several times over the last 10 years for its software-development practices.
The FDA's policy is to inspect any company that makes medical devices about every two or three years. Each time its examiners visited Multidata they found problems. The FDA would not say how common it is to find problems on each company visit, but Gladys Rodriguez, head of the FDA enforcement unit that deals with medical devices and radiological health products, says an injunction is a "rare" action.
According to documents obtained by Baseline under the Freedom of Information Act, the FDA inspected Multidata's products and manufacturing operations four times in the past dozen years.
In 1993, 1995 and 1998, FDA inspectors found the same deficiencies in Multidata's software-development process coming up time and again: a lack of good software specification and documentation procedures to guide and control the software-development and change process; insufficient documentation to show that the software had been properly tested to see if it worked; and inadequate investigation into customer complaints.
The FDA inspectors' report from the 1993 investigation noted: "[C]omplaint review found a number of reports of software errors or 'bugs' which indicate Multidata's software testing is incomplete. Several of these complaints reported incorrect dose calculation described as 'off by about 20%', and 'bizarre' and 'dramatic.' Follow-up investigation of these and other complaints found many software errors present in software shipped to customers that could have been found with structured, thorough, and rigorous testing throughout the software development process using basic software analysis and testing techniques."
Conley says the company looked into the complaints and corrected any problems. "We fixed what we found," he says, adding that some of the complaints the company looked into were not software-related, but resulted from users being unfamiliar with its product. And, he says, there's no link between the 1993 report and the accidents in Panama.
After the FDA's 1998 inspection, the agency sent a warning letter to the company, outlining the findings from its review and instructing the company to look into its software-development process and correct the deficiencies noted in its report. The agency told Multidata that if the company didn't take care of the problems, the FDA would be forced to take further action, which could include fines and an injunction against the company.
The FDA also told Multidata it wanted the company to notify the agency within 15 days of the steps the vendor was going to take to address the problems.
According to the subsequent FDA reports, Multidata never responded to the 1998 letter.
"We usually get action from our warning letters," Rodriguez says. Multidata's Roestel admits that the company did not respond to the FDA's letter.
In 2001, after the deaths in Panama, FDA inspectors found many of the same problems they found earlier. Multidata, it said, had no mechanisms for addressing incomplete or ambiguous software requirements, customer complaints were not being properly recorded, and there was no comprehensive testing plan to demonstrate that its software was "fit for use."
Conley says the company has been working hard to rectify the complaints identified in the FDA's 2001 inspection. He says Multidata, in fact, was looking for ways to better track software fixes and problems, including the installation of a computerized system to record and store customer complaints, when the FDA issued its injunction.
Conley admits Multidata didn't address issues quickly enough for the FDA.
"We're a small company that didn't always react in a timely fashion," he says. "We did what we were supposed to do, [but] we didn't file the proper reports for it."
To have the injunction lifted, the FDA says Multidata must improve its design and manufacturing methods; upgrade its record-keeping mechanisms; and retain a medical-device design expert to inspect the company's manufacturing activities, check over its software code, and report back to the FDA. The outside expert must have no financial ties to the company other than the consulting agreement for this series of tasks.
Conley said in February 2004 that Multidata will have its development practices reviewed by Bio-Reg Associates Inc., which describes itself as a "regulatory consulting firm conveniently located close to the FDA in Washington, D.C.''
"They are staking their reputation on the line by representing us to the FDA," Conley says. "They would not take us if we were a shlock outfit."