Security Case: How To Survive a Bot Attack

It started early on a Sunday afternoon, with calls to the help desk at Seattle’s Northwest Hospital & Medical Center. PCs were running slow, the hospital staff complained, and documents wouldn’t print. On Monday morning, Jan. 10, 2005, as more employees came to work and logged on to their systems, the problems spread.

By 10 a.m., all 50 people in the hospital’s information-technology department were summoned. But their efforts made little difference. Strange things started happening. Operating-room doors stopped opening, and doctors’ pagers wouldn’t work, according to the FBI, which later investigated. Computers in the intensive care unit shut down.

“I could see everybody was very frightened,” says Robert Steigmeyer, Northwest’s chief financial officer. “You saw the worry and concern in everybody’s eyes.”

As the hospital’s information-technology staff would soon discover, Northwest was under attack by a botnet, a network of PCs infected with code that was controlled, in this case, by a 19-year old Californian, Christopher Maxwell, and two juveniles. The trio exploited a flaw in Microsoft Windows that let them install pop-up ads—adware—on the hospital’s computers. They got into the first one that Sunday. As the bad code coursed through the network, the hospital’s computers started turning into bots, too, expanding Maxwell’s stable of zombie PCs. These new bots in turn scanned the network, looking for new victims to infect, and the network clogged with traffic. Hospital communications began to break down.

Northwest wasn’t Maxwell and crew’s only prey. Among their other victims from July 2004 to July 2005 were the Department of Defense and Colton Joint Unified School District in California, according to court papers.

In May 2006, based in part on evidence supplied by the hospital, Maxwell pleaded guilty to conspiracy and intentionally causing damage to a protected computer. He was sentenced in August to 37 months in federal prison.

At the hospital, the attack’s aftermath lasted for weeks. As computers stopped working, hospital workers relied on backup systems—people and paper. Extra workers were brought in to help carry out tasks by hand. Lab results, for instance, were run from the lab to the hospital floor to the patient’s bedside. To save time, elective procedures were postponed. Every day, department managers would meet to make sure the new routines were holding and no patients were endangered.

“The first week was incredibly disruptive,” Steigmeyer says. “Patient safety was very high on everybody’s list.” He adds that patient data wasn’t compromised during the attack.

Initially, Northwest’s tech team tried to halt the attack by shutting off the hospital from the Internet. Even though the bots were now contained internally, they still infected PCs faster than the team could clean them, says Ken Burton, Northwest’s chief technology officer.

By Monday afternoon, the I.T. department figured out which malware the bots were installing on PCs and wrote a script, which was pushed out hourly, directing PCs to remove the bad code. This “helped quiet things down a bit,” Burton says.

By Tuesday, Computer Associates—Northwest’s antivirus vendor—figured out which malware Maxwell had used to get into the network and wrote a virus signature that blocked new code from coming in. The attack was contained to 150 of 1,000 PCs—all of which had to have their hard drives wiped clean and their software reinstalled, at an estimated cost of $150,000. Maxwell has been ordered to pay back about 75% of that amount.

The hospital’s network is now protected by CA’s Pest Patrol, which blocks adware and spyware, and Cisco MARS, an intrusion detection system. Northwest’s I.T. staffers no longer wait for vendors, particularly Microsoft, to certify software patches before applying fixes—they evaluate and test patches themselves. The Windows flaw that the attack slipped through had not yet been patched on the hospital’s PCs. The servers had been patched and so escaped.

While the FBI says many organizations avoid discussing attacks because they fear a loss of business, Steigmeyer says the hospital wanted to speak up on behalf of institutions that have vulnerable public infrastructure.

As Steigmeyer puts it: “We believe we raised awareness of the potential dangers of an attack.”