Unlike the Sept. 11 terror attacks and the Northeast power outage in August 2003, Hurricane Katrina gave companies throughout the Gulf Coast time to back up data and implement their business continuity plans.
The only question was whether companies prepared properly for Katrina’s wrath. Backup tapes, data center redundancy and temporary headquarters are one thing. It’s another to lose power and roads, plants, the supply chain and the business ecosystem. The rub with disaster recovery planning is that you have to make assumptions and balance risks with costs. Should a Manhattan company prepare for a Category 5 hurricane? Does a Des Moines, Iowa, firm prep for a nuclear bomb? How far away should a backup data center be from a primary? Twenty-five miles? Twenty-five-hundred?
There are no easy answers. But as PSS/World Medical and Harrah’s found out, proper assessment of a company’s physical and digital resources, as well as a system to track human resources, can prove invaluable in a disaster.
Estimate Potential Losses
PSS/World, a Jacksonville, Fla., medical equipment distributor, doesn’t prepare for any one catastrophic event. Instead, the company in 2003 created a plan that it would follow regardless of the damage done to any of its facilities. This included its Jackson, Miss., call center, which sits 120 miles north of the Gulf Coast but was still shut down by Katrina.
Katrina and Rita didn’t just wipe out data centers; they wiped out a lot of disaster plans, too. Check out:
The first step the company took when it was formulating its plan was to calculate potential dollars lost.
If a natural or manmade disaster prevented it from shipping equipment to nursing homes and hospitals, the disaster would cost the company about $3 million a day. That potential loss was then weighed against the $500,000 a year it costs to use disaster recovery services from SunGard Availability Services. “Let’s just say this wasn’t a hard sell for our CEO,” says chief technology officer Brian Finley. PSS/World stores its data in the vendor’s Philadelphia facility.
The baseline of $3 million was determined the hard way. In November 2003, PSS/World’s storage array system, which houses critical customer and inventory data, malfunctioned and shut down the entire company for more than 20 hours. Finley didn’t go into specifics, saying only that it was a critical mistake made by the vendor installing the storage system.
As Katrina churned toward the Mississippi coast, PSS/World alerted SunGard on Aug. 28 that it wanted to put its contingency plan into effect. If the storm knocked out PSS/World’s call center for a week, that would be $21 million in lost sales, or 5.4% of its first-quarter sales of $387.1 million.
PSS/World then sent 50 employees to SunGard’s crisis management center in Atlanta to manage the company’s databases, network and call center. The Atlanta center can link to the network servers at the Philadelphia site, allowing folks in Atlanta to update files and process orders. The SunGard sites also allow PSS/World to forgo tape backups because all the data is hosted at those facilities, saving the company 36 hours of recovery time, or $4.5 million, Finley says.
Story Guide:
