Katrina ArrivesBy David F. Carr | Posted 2005-10-04 Email Print
WEBINAR: On-demand webcast
Next-Generation Applications Require the Power and Performance of Next-Generation Workstations REGISTER >
Vacuum-cleaner maker Oreck had the linchpin of its business continuity plan blown away by Hurricane Katrina. Intercosmos Media Group rode out Katrina with half a terabyte of data on the line, guarded by a former Green Beret whose preparations included gen
Aug. 29: Katrina makes landfall east of New Orleans at 6 a.m. Central Daylight Time, packing Category 4 winds between 131 and 150 miles per hour.
"I'm no meteorologist, but it looks to me like this storm is falling apart fast as it hits land."
Barnett's post at 4:20 a.m. on the day Katrina makes landfall shows how the Big Easy exhaled as Katrina veered away from the heart of New Orleans. Barnett was optimistic. "When you look at the damage New Orleans sustained vs. what we were told to expect, you're left with the impression that they will never be able to talk anyone into evacuating again," he writes in his blog.
By 9:30 a.m., though, DirectNIC has lost power amid hurricane-shattered windows. With rain whipping through the windows, Solares uses
T-shirts from a failed business venture to keep water from seeping under the doors of the server room.
Over at Oreck, executives are operating out of a Houston hotel room. At the company's Boulder site, Oreck employees are loading backup tapes with the data that will be used to fulfill orders at the Long Beach site—assuming it is functional. The company's call center in Denver is also operational. Oreck gets early reports from Long Beach. "Long Beach was devastated all around the plant," Oreck says. "Whole neighborhoods were gone. There were propane tanks from houses in trees."
The destruction at Long Beach meant the linchpin of Oreck's disaster recover plan—the 76 miles between Long Beach and New Orleans—was gone. "I don't think any of us anticipated a single storm taking out both locations," Oreck says.
In hindsight, it's clear that Oreck's facilities weren't far enough apart, but companies have to balance the reality of getting workers to a location versus distance, says Dave Luce, president of the Society for Information Management, a national organization of chief information officers, and CIO of the Rockefeller Group, a real estate operating company based in New York. After Sept. 11, Luce's company created a co-location center 25 miles away in New Jersey. "We wanted it outside New York City but still be accessible to our workforce," Luce says. "You could argue 25 miles is too close, but disaster recovery isn't scientific."
Aug. 30: New Orleans' levees between the 17th Street Canal leading to Lake Pontchartrain break, flooding the southern and western parts of downtown.
"The infrastructure required to maintain a city is down," Barnett reports. "It could be a long time before it's back up. There will be too many people fighting for exceptionally scarce resources. It's one of those situations where you need A in order to fix B, but you can't do A until C happens and C can't happen until B is finished."
New Orleans is flooded. DirectNIC's data center is still dry, but so is the plumbing. With no running water, Barnett uses his Army training to institute water conservation and emergency hygiene measures. He admits he "never trained for the total collapse of civilization."
Barnett's biggest concern: stretching the capacity of the data center's diesel generators, which requires moving and dumping 55-gallon drums of diesel all day. DirectNIC has 10 days of fuel, but Barnett wonders if it will last. Another issue: safety. Just before midnight, the DirectNIC team blocks building entrances to deter looters.
In Houston, Tom Oreck sees New Orleans' levees break on television and knows a quick return to headquarters is unlikely. "We knew then it was not a matter of days, but a matter of months," he says.
The news from the Long Beach facility, about a mile and a half from the beach, was better. An employee got into the 375,000-square-foot plant to find it "damaged but intact and fixable," Oreck says. The sign on the front of the building is gone, but overall the factory shows few signs of damage. Across the railroad tracks toward the beach, homes were swept off their foundations.
This day required two tasks for Oreck.
First, find employees since 30% of Oreck's workers have lost their homes. The company has accounted for all but 100 of its 1,500 employees. To find the missing, Oreck sets up an 800 number and rudimentary Web site to collect employee locations. "Finding our people was the most important piece," Oreck says. "People need shelter, food, water and help with insurance claims to help them move toward normalcy. Good business, its systems and facilities are the result of good people."
Second, Oreck had to boil the company down to its most critical functions. "We had to look at this business from a fundamental 40,000-foot view," he explains. "What did we need to continue?"
The answer: Manufacturing capability, distribution, a call center and the back office operations, notably human resources, payroll and accounts payable and receivable. The computer systems given priority were the ones tied to those functions—call center operations in Denver, accounting and human resources in Dallas, and supply chain systems in Boulder. Project teams of three to five people were assigned to the categories.
Direct sales through Oreck's Web site were shut down in favor of support for the company's 450 retail stores. Point-of-sale systems in company-owned stores were shut down in favor of paper receipts, and a planned consolidation of those systems across company-owned and franchised stores was put off. And a move from a supply chain system installed in 1997 to SAP for Retail was delayed.