Disaster Planning: How Much Makes Cents?

 
 
By Baselinemag  |  Posted 2005-10-04
 
 
 

Unlike the Sept. 11 terror attacks and the Northeast power outage in August 2003, Hurricane Katrina gave companies throughout the Gulf Coast time to back up data and implement their business continuity plans.

The only question was whether companies prepared properly for Katrina's wrath. Backup tapes, data center redundancy and temporary headquarters are one thing. It's another to lose power and roads, plants, the supply chain and the business ecosystem. The rub with disaster recovery planning is that you have to make assumptions and balance risks with costs. Should a Manhattan company prepare for a Category 5 hurricane? Does a Des Moines, Iowa, firm prep for a nuclear bomb? How far away should a backup data center be from a primary? Twenty-five miles? Twenty-five-hundred?

There are no easy answers. But as PSS/World Medical and Harrah's found out, proper assessment of a company's physical and digital resources, as well as a system to track human resources, can prove invaluable in a disaster.

Estimate Potential Losses

PSS/World, a Jacksonville, Fla., medical equipment distributor, doesn't prepare for any one catastrophic event. Instead, the company in 2003 created a plan that it would follow regardless of the damage done to any of its facilities. This included its Jackson, Miss., call center, which sits 120 miles north of the Gulf Coast but was still shut down by Katrina.

Katrina and Rita didn't just wipe out data centers; they wiped out a lot of disaster plans, too. Check out:Baselinemag.com and CIOInsight.com's coverage of the disasters, and the efforts of CIOs to save the data on which their companies depend.

The first step the company took when it was formulating its plan was to calculate potential dollars lost.

If a natural or manmade disaster prevented it from shipping equipment to nursing homes and hospitals, the disaster would cost the company about $3 million a day. That potential loss was then weighed against the $500,000 a year it costs to use disaster recovery services from SunGard Availability Services. "Let's just say this wasn't a hard sell for our CEO," says chief technology officer Brian Finley. PSS/World stores its data in the vendor's Philadelphia facility.

The baseline of $3 million was determined the hard way. In November 2003, PSS/World's storage array system, which houses critical customer and inventory data, malfunctioned and shut down the entire company for more than 20 hours. Finley didn't go into specifics, saying only that it was a critical mistake made by the vendor installing the storage system.

As Katrina churned toward the Mississippi coast, PSS/World alerted SunGard on Aug. 28 that it wanted to put its contingency plan into effect. If the storm knocked out PSS/World's call center for a week, that would be $21 million in lost sales, or 5.4% of its first-quarter sales of $387.1 million.

PSS/World then sent 50 employees to SunGard's crisis management center in Atlanta to manage the company's databases, network and call center. The Atlanta center can link to the network servers at the Philadelphia site, allowing folks in Atlanta to update files and process orders. The SunGard sites also allow PSS/World to forgo tape backups because all the data is hosted at those facilities, saving the company 36 hours of recovery time, or $4.5 million, Finley says.

Story Guide:

Diary of Disaster: Riding Out Katrina in the Data Center

Your People Come First

While PSS/World sent 35 of its Jackson employees to Atlanta before the storm hit, employees are usually in the area when a disaster strikes. And if a hurricane or other event takes down a company's buildings or communications center, people can lose touch.

Katrina, for instance, destroyed two Harrah's Entertainment riverboat casinos docked in Gulfport and Biloxi, Miss.

Harrah's created a company intranet within two days that allowed employees to access emergency relief information as well as begin finding work at other Harrah's properties throughout the U.S. The intranet was instrumental in helping Harrah's locate all of its employees in the affected area. Companies such as Oreck set up toll-free information lines.

"You evaluate every risk, but the most urgent thing for employees is finding their families and coworkers," says M. Lewis Temares, vice president of information technology and dean of the College of Engineering at the University of Miami, which shut down for four days because of Katrina's pass through that city.

Alberto Lopez, a spokesman for Harrah's, says no plan could have dealt with the damage that Katrina inflicted on the casinos, which cost roughly $200 million to replace. "This isn't something you can prepare for," he says.

Story Guide:

Diary of Disaster: Riding Out Katrina in the Data Center