Witnessing Chaos

 
 
By David F. Carr  |  Posted 2005-10-04
 
 
 

"We're on the 10th and 11th floor of a corporate high rise on Poydras Ave., right near St. Charles. We have generators and tons of food and water. It is five of us total. I am not sure how the Internet connection will be affected. I have a camera and my gun. Sustained winds are 175, gusts to 215. The real danger is not the wind, it's the storm surge the wind will be pushing into the city from the Gulf through the lake. The city might never recover. Honestly, this thing could be biblical"

That online diary, or weblog, entry was posted about noon on Sunday, Aug. 28, by Michael Barnett, a former Green Beret and business consultant to Intercosmos Media Group, the parent company of domain registrar DirectNIC and Zipa.com, a Web host. Barnett, a lifelong friend of Intercosmos chief executive officer Sigmund Solares, was holed up with his girlfriend Crystal Coleman in a data center in New Orleans' Central Business District, an alleged safe place to ride out Hurricane Katrina. Solares hired Barnett as crisis manager for the storm.

Barnett updated his blog during the nation's greatest natural disaster, as wind, rain, floods, looting and death engulfed one of America's most beloved cities. Research firm Economy.com expects economic losses of $125 billion.

According to the U.S. Census Bureau, New Orleans is home to 30,262 businesses, including little-known companies like Intercosmos and well-known ones such as vacuum cleaner manufacturer Oreck Corp. The storm tested both companies' business continuity strategies.

Intercosmos' plan: Back up critical data from its New Orleans data center remotely, but, because of cost, gamble with half a terabyte of Web content from thousands of free hosting accounts bundled with DirectNIC.

Oreck's plan: Move operations from its headquarters and data center in New Orleans to its Long Beach, Miss., manufacturing and call center facility, and vice versa. These operations would be supplemented by a backup data center in Boulder, Colo., and a call center in Denver. Oreck didn't plan on both its New Orleans and Long Beach facilities taking a hit.

The lesson: Plan for a worst-case scenario, but remember that disasters have the potential to exceed expectations.

Here is an account of how these two companies struggled to keep their businesses, people and technology running before and after Katrina blew apart the Gulf Coast. The efforts stress the importance of backing up key data and people in dispersed locations, of being able to strip your company down to its essentials, and of having a staff that can improvise even as the best-laid plans go awry.

Aug. 27, 28: Katrina builds in the Gulf of Mexico and becomes a Category 5 hurricane with winds of 175 mph.

Katrina and Rita didn't just wipe out data centers; they wiped out a lot of disaster plans, too. Check out: Baselinemag.com and CIOInsight.com's coverage of the disasters, and the efforts of CIOs to save the data on which their companies depend.

Intercosmos moves some critical functions such as DirectNIC name servers, which are required to keep its clients' domain names functioning, to backup locations in Florida and Atlanta.

CEO Solares, however, keeps half a terabyte of customer and Web content data in New Orleans. That data—supporting the free, basic Web sites customers get with their $15 domain registration—could be restored from backup tapes, if necessary, but at the low prices DirectNIC charges, it can't afford full redundancy. Besides, these customers don't get the same guarantees as if they were paying for their sites through Zipa.com, the company's Web hosting subsidiary.

Still, to protect the company's reputation with its customers, he wants to keep sites online.

Meanwhile, Oreck backs up its last eight hours of activity—invoices, manufacturing requests and shipment information—on tapes to be loaded on its IBM AS/400 mainframes in Boulder. Oreck CEO Tom Oreck, the 54-year-old son of company founder David Oreck, takes the tapes to Houston and sends them to Boulder via Federal Express.

CEO Oreck then launches the company's contingency plan, which transfers the company's call center to Denver, a site that takes call overflow from its Long Beach call center. Oreck employees take all the data and documents—customer data, shipping destinations and inventory information—the company would need to restart later.

Story Guide:

Diary of Disaster: Riding Out Katrina in the Data Center

Aug. 29: Katrina makes landfall east of New Orleans at 6 a.m. Central Daylight Time, packing Category 4 winds between 131 and 150 miles per hour.

"I'm no meteorologist, but it looks to me like this storm is falling apart fast as it hits land."

Barnett's post at 4:20 a.m. on the day Katrina makes landfall shows how the Big Easy exhaled as Katrina veered away from the heart of New Orleans. Barnett was optimistic. "When you look at the damage New Orleans sustained vs. what we were told to expect, you're left with the impression that they will never be able to talk anyone into evacuating again," he writes in his blog.

By 9:30 a.m., though, DirectNIC has lost power amid hurricane-shattered windows. With rain whipping through the windows, Solares uses

T-shirts from a failed business venture to keep water from seeping under the doors of the server room.

Over at Oreck, executives are operating out of a Houston hotel room. At the company's Boulder site, Oreck employees are loading backup tapes with the data that will be used to fulfill orders at the Long Beach site—assuming it is functional. The company's call center in Denver is also operational. Oreck gets early reports from Long Beach. "Long Beach was devastated all around the plant," Oreck says. "Whole neighborhoods were gone. There were propane tanks from houses in trees."

The destruction at Long Beach meant the linchpin of Oreck's disaster recover plan—the 76 miles between Long Beach and New Orleans—was gone. "I don't think any of us anticipated a single storm taking out both locations," Oreck says.

In hindsight, it's clear that Oreck's facilities weren't far enough apart, but companies have to balance the reality of getting workers to a location versus distance, says Dave Luce, president of the Society for Information Management, a national organization of chief information officers, and CIO of the Rockefeller Group, a real estate operating company based in New York. After Sept. 11, Luce's company created a co-location center 25 miles away in New Jersey. "We wanted it outside New York City but still be accessible to our workforce," Luce says. "You could argue 25 miles is too close, but disaster recovery isn't scientific."

Aug. 30: New Orleans' levees between the 17th Street Canal leading to Lake Pontchartrain break, flooding the southern and western parts of downtown.

"The infrastructure required to maintain a city is down," Barnett reports. "It could be a long time before it's back up. There will be too many people fighting for exceptionally scarce resources. It's one of those situations where you need A in order to fix B, but you can't do A until C happens and C can't happen until B is finished."

New Orleans is flooded. DirectNIC's data center is still dry, but so is the plumbing. With no running water, Barnett uses his Army training to institute water conservation and emergency hygiene measures. He admits he "never trained for the total collapse of civilization."

Barnett's biggest concern: stretching the capacity of the data center's diesel generators, which requires moving and dumping 55-gallon drums of diesel all day. DirectNIC has 10 days of fuel, but Barnett wonders if it will last. Another issue: safety. Just before midnight, the DirectNIC team blocks building entrances to deter looters.

In Houston, Tom Oreck sees New Orleans' levees break on television and knows a quick return to headquarters is unlikely. "We knew then it was not a matter of days, but a matter of months," he says.

The news from the Long Beach facility, about a mile and a half from the beach, was better. An employee got into the 375,000-square-foot plant to find it "damaged but intact and fixable," Oreck says. The sign on the front of the building is gone, but overall the factory shows few signs of damage. Across the railroad tracks toward the beach, homes were swept off their foundations.

This day required two tasks for Oreck.

First, find employees since 30% of Oreck's workers have lost their homes. The company has accounted for all but 100 of its 1,500 employees. To find the missing, Oreck sets up an 800 number and rudimentary Web site to collect employee locations. "Finding our people was the most important piece," Oreck says. "People need shelter, food, water and help with insurance claims to help them move toward normalcy. Good business, its systems and facilities are the result of good people."

Second, Oreck had to boil the company down to its most critical functions. "We had to look at this business from a fundamental 40,000-foot view," he explains. "What did we need to continue?"

The answer: Manufacturing capability, distribution, a call center and the back office operations, notably human resources, payroll and accounts payable and receivable. The computer systems given priority were the ones tied to those functions—call center operations in Denver, accounting and human resources in Dallas, and supply chain systems in Boulder. Project teams of three to five people were assigned to the categories.

Direct sales through Oreck's Web site were shut down in favor of support for the company's 450 retail stores. Point-of-sale systems in company-owned stores were shut down in favor of paper receipts, and a planned consolidation of those systems across company-owned and franchised stores was put off. And a move from a supply chain system installed in 1997 to SAP for Retail was delayed.

Story Guide:

Diary of Disaster: Riding Out Katrina in the Data Center

Aug. 31: Looting becomes rampant in New Orleans.

"Right now we're trying to show you all the looting," Barnett reports, providing some narration to go with the video he's broadcasting via Webcam. "Guys pushing shopping carts with 40 Nike boxes in them. People breaking into cars. Assaulting ATM machines. It's hard just to sit by and do nothing."

Barnett manages to reduce the draw on the generator from 30% capacity to 20%. The DirectNIC team answers a call for help from another nearby firm that wants to get access to the data on its servers. The data center team fetches the other company's servers, powers them up and assists with a 40-gigabyte download of corporate data. In return, they get 25 gallons of bottled water and assorted cleaning supplies.

Meanwhile, Oreck's project teams have to plan to start a Long Beach plant with no electricity, no generators and no infrastructure to distribute vacuum cleaners, air purifiers and cleaning products. First, Oreck corrals generators for power and trailers for employees, but still can't print out bar codes, track orders and ship its products without network connections.

To help with logistics, it calls United Parcel Service. Oreck, a longtime customer of UPS for shipping, needs help printing bar codes, picking order tickets and distributing products as if they came from the Long Beach plant.

After consulting with the company's UPS account representative, Richard Behrendt, and other UPS executives, Oreck agrees on a fix. The plan: Allow UPS to tap into Oreck's mainframe systems in Boulder to grab orders and serial numbers and print them out in Atlanta. This information would be attached to products manufactured in Long Beach. UPS would truck in food and water for employees and then leave with Oreck products from UPS' Atlanta facility.

Such an arrangement typically takes six to eight weeks, but Oreck wants to start the Long Beach site by Sept. 9, or five work days. UPS doesn't know Oreck's supply chain system and needs to find people internally who can navigate "green screens" and mainframes. Oreck's mainframe-based supply chain system lacks a manual and customized function keys. Two Oreck employees train the UPS teams on using special commands to hop from one screen, say, pick ticket, to another, such as shipping destination. Labor is taken from other UPS shifts and temporary workers.

Sept. 1: Conditions in the city unravel as police are overwhelmed.

"The coroner's office is shut down so bodies are being covered in leaves at best or left where they lie at worst. Until we get a military presence of significance in the city, the roving gangs of thugs own the streets."

An exhausted New Orleans police officer takes shelter in the DirectNIC offices. The police command and control structure is so shot that the officer didn't know New Orleans is under martial law.

At 9 a.m., Barnett reports that the Federal Emergency Management Agency and the city will permit DirectNIC to get diesel shipped in if someone can deliver it. By 10:30 a.m., a fuel truck is waiting downstairs. The problem? The truck arrives before the data center team can get the fuel drums downstairs. The driver leaves.

At this juncture, DirectNIC 's high-speed Internet connections are failing as backup power for telecommunications substations around the city fades, making it necessary for DirectNIC to ditch some Web sites and services. One casualty: Something Awful, a humor Web site, which is hosted at rock-bottom rates and consumes too much bandwidth. When he took on the account, Solares cut a deal that allows for the site to be dropped in a crisis.

Outside, the situation worsens. Dead bodies in the streets and disorganized relief efforts prompt Barnett to send a distress signal: "In case anyone in national security is reading this, get the word to President Bush that we need the military in here NOW. The city is being lost .... We need the kind of logistical support and infrastructure only the Active Duty military can provide."

That afternoon, the diesel delivery succeeds on the second try. Solares tries to contact his company's telecommunications providers, BellSouth and TelCove, to revive connections. Meanwhile, Barnett and Solares wade through filthy water to supply City Hall with an Internet router for emergency communications.

That night, Solares tries to contact employees now working remotely to say he plans to pay them whatever they earned in their last pay period, whether they've worked or not. However, Solares can't reach the accounting staffers responsible for payroll.

For Oreck, the next four days are spent procuring trailers for homeless employees, lining up generators for its Long Beach plant and finding its workers. CEO Oreck commutes between his Houston hotel room and the company's temporary facility in Dallas.

Sept. 2: Military arrives in New Orleans to bring aid and restore order.

"This place is completely coming apart. The hopelessness on the street breaks the heart. The old, the tired, the sick seem resigned to their presumed fate. Death."

Looking down on New Orleans from the roof, 27 stories above the ground, Barnett senses the city will never be the same. He also doesn't like the way "stressed-out, trigger-ready police and military types" look at him while he's waiting for a fuel delivery.

Noah Lieske, a co-owner of Intercosmos, adds a blog post requesting a place for 55 people to work temporarily, ideally with 6,000 to 8,000 square feet of office space and a data center nearby.

Meanwhile, Oreck is on the move. With Oreck's New Orleans location inoperable for an undetermined time, executives and workers representing technology, payroll and human resources populate a 20,000-square-foot IBM disaster recovery site in Dallas with 100 workstations. CEO Tom Oreck, a longtime IBM customer, procured the space after the levees broke in New Orleans, but hopes the company's stay doesn't last too long.

"We're fully functional there, but it is close quarters and there's only so long we can do that," Oreck says. "If we can get back to our facility in the next 60 to 90 days, we'll be OK. If longer, we'll have to find other arrangements in Dallas or Houston—temporary sites for the longer term." 8

Story Guide:

Diary of Disaster: Riding Out Katrina in the Data Center

; Evacuations Continue">

Sept. 2 to 5: New Orleans stabilizes. Evacuations continue.

"The government is never equipped to handle a crisis like this. There's too much bureaucracy —initiative-stifling bureaucracy, which prevents swift, effective action."

It's Labor Day weekend, and DirectNIC is down to the last of its four high-speed Internet connections from TelCove, which isn't sure it can get enough diesel fuel to its generators before the connection drops.

With the help of Brian Acosta of I-55.com, DirectNIC's Internet service provider, the company helps TelCove get its fuel and restore two other Internet connections before the end of the day. Web sites that were temporarily dropped get brought back online.

Solares works with the city government and police to help those agencies communicate. The city has been using voice-over-Internet protocol phones, and he's offering equipment and expertise.

By Labor Day, all four of DirectNIC's Internet connections are online, putting the company solidly back in business, even though it's still running on generator power. Other buildings in the Central Business District are starting to get back power.

For Oreck, generators and trailers are secured as the company plans for a plant reopening on Sept. 9.

Sept. 6: Evacuations finish. Water in the city begins to recede.

"Man, it would be nice not to have to haul diesel anymore."

Barnett is making plans to pull servers from the Pan Am Life building across the street and get them online so Pan Am can download data to a backup location. DirectNIC's staff has already secured servers for other businesses in the area, bartering this service for water and other supplies.

Labor Day weekend is over, and the folks at Oreck and UPS are relieved. The long weekend stalled Oreck's efforts to truck its generators to its facilities, not to mention the trailers to house some of its homeless employees in Long Beach.

UPS managers go to Long Beach to get some of Oreck's equipment, notably the configured Zebra Technologies printers to create bar codes in the vacuum cleaner company's format. UPS also has to install electric wiring, network connections and lighting in a vacant area of the parcel company's Atlanta facility to absorb Oreck's shipments.

UPS delivers food and water to Oreck, which then fills the truck with orders that lack bar codes and serial numbers. UPS gets a pallet of products, say, air purifiers. From there, workers take apart the pallet and create a pick ticket or directions to bundle orders scheduled for a destination, such as Langhorne, Pa. An invoice is then recorded and shipped. Data is recorded manually into the Oreck mainframe in Boulder so customers can receive tracking information from call center representatives. "I've been the project manager for a lot of projects," says John Bunker, client solutions manager for the eastern district of UPS Supply Chain Solutions. "But this one was not much about planning, just executing."

Twenty-four to 30 trucks would make trips from Atlanta to Long Beach between Sept. 6 and 9, according to Behrendt.

Sept. 7 to 8: New Orleans calms, becomes a military town.

"If you want to play soldier with me, I will make you play it a lot longer than you had in mind."

Sometime after midnight on Sept. 8, a squad of 82nd Airborne soldiers accompanied by a U.S. Marshal bust into Intercosmos' data center, saying they're investigating lights and movement in the building. Annoyed at the intrusion, Barnett tells them that if they think there are intruders, they'd better check the building from top to bottom. His reward: Knowing that the building has been inspected, he gets his best sleep in weeks.

In the morning, Coleman leaves along with Solares and Donny Simonton, a senior vice president and technology manager. They're replaced by three fresh employees, while Solares scouts for new office locations.

Meanwhile, New Orleans Mayor C. Ray Nagin has issued a mandatory evacuation order, but the DirectNIC people think they're exempt until told otherwise.

At Long Beach, Oreck ships products that were ordered before Katrina hit but never distributed. Employees reappear at the plant, which is gearing up for an opening ceremony with a cookout and appearance from Mississippi. Gov. Haley Barbour.

Sept. 9: New Orleans improves, but much of the city is still under water.

"I work well alone or in a group, I am self-motivated, I am an idea man, I pay attention to detail, and I'm not afraid to get my hands dirty."

Barnett, still working at the DirectNIC site, is looking for another gig. In the next few days, he hopes to stop hauling diesel and fretting about generator maintenance.

Oreck opens its Long Beach facility. Although the plant opens, it's not the same. The roof leaks. Information systems remain out due to a lack of communications infrastructure, and just two of five manufacturing lines are open. Oreck is manufacturing, shipping and receiving with pen and paper rather than computers driving processes. Power is back on but the phone lines are still down, leaving the factory cut off from its Dallas data center.

Electronic communication with the outside world has been reduced to executives calling suppliers from beneath the crumpled metal canopy of an outside break area, the spot where cell phone reception is best. Oreck supply chain director Candi Mauffray, however, says the factory is getting by, with 300 of 500 employees and 80% of the salaried staff back at work. "The one thing we don't have is systems," she says.

Indeed, at a Langhorne, Pa., company-owned Oreck outlet, the following sign sits at the checkout counter: "Please note our computer system is 'down' indefinitely. This is due to the fact that Oreck's corporate offices are located in New Orleans, La. We appreciate your patience and understanding as we provide written receipts."

Story Guide:

Diary of Disaster: Riding Out Katrina in the Data Center

Postscript

"Our building got power right at noon central time, and we had to manually reset the generator and get power to switch from the batteries to the city grid. That took about 5 minutes. NO MORE DIESEL DUTY."

On Sept. 12, DirectNIC got its power back, but it remains to be seen what happens to the company's location. Solares plans to keep the New Orleans data center running with a skeleton crew because he has favorable lease terms. Whether the company headquarters will come back is another question, though. At the least, Solares says he'll need better backup capability elsewhere.

On Sept. 14, cleanup crews at Oreck's Long Beach plant battle mold from storm water. Oreck's Web site is now running again, but there are two- to three-week delays on Oreck.com orders. The main challenge for Oreck is communications. BellSouth cannot give a timeline on laying the lines it needs to restore communications.

CEO Oreck is looking at other temporary locations for his facilities in Dallas in case the company can't move back to New Orleans quickly. For Oreck, the company's return to New Orleans is a matter of when, not if. "The only reason we wouldn't go back is if New Orleans was a dead city," he says. "Short of that, New Orleans is our home and where our families are. I never seriously considered not coming back."

Meanwhile, New Orleans suffered additional flooding on Sept. 24 from Hurricane Rita, a Category 3 storm, that hit the Texas and Louisiana coasts.

Katrina and Rita didn't just wipe out data centers; they wiped out a lot of disaster plans, too. Check out:Baselinemag.com and CIOInsight.com's coverage of the disasters, and the efforts of CIOs to save the data on which their companies depend.

Lessons Learned

No matter what the disaster, there are common items that shouldbe in business continuity plans.

Here are Katrina's lessons.

PEOPLE FIRST. The first effort should be focused on locating your employees and ensuring their safety.

DISTANCE MATTERS. Put your backup data center in a locale far enough away from your primary center to ensure continuity, but close enough to get employees there.

PRIORITIZE YOUR BUSINESS. Focus on the information systems that matter most, such as customer support and manufacturing.

EXPECT THE UNEXPECTED. No matter how well documented your business

Continuity plan, the disaster is likely to throw you an unexpected curveball. Account for the worst possible scenario.

While few companies have said they will permanently leave New Orleans, some firms, such as Ruth's Chris Steak House, won't return.

What to Consider:

Plans Already in the Works. On Sept. 6, Ruth's Chris Steak House, founded in 1965 by New Orleans native Ruth Fertel, said it was permanently relocating to Orlando, Fla., from Metairie, La., a New Orleans suburb. But in the company's Aug. 10 initial public offering prospectus, the company said it expected to relocate corporate headquarters to a new facility "within the next 24 months."

Replacing Workers Who Won't Move. Ruth's Chris CEO Craig Miller said 60% of

its workers at headquarters accepted relocation packages to Florida and 40%, mostly clerical and accounting workers, declined. Factor in whether you could replace skilled workers before making a move.

Incentives to Stay Put. To be sure, tax incentives are a big component of any decision.

New York City and the state of New York are still offering tax write-offs to companies that relocate to the World Trade Center area following the Sept. 11 attacks.

Calculate: How Much Would It Cost You to Move Your Data Center?

Story Guide:

Diary of Disaster: Riding Out Katrina in the Data Center