Diary of Disaster: Riding Out Katrina in the Data CenterBy David F. Carr | Posted 2005-10-04 Email Print
Re-Thinking HR: What Every CIO Needs to Know About Tomorrow's Workforce REGISTER >
Vacuum-cleaner maker Oreck had the linchpin of its business continuity plan blown away by Hurricane Katrina. Intercosmos Media Group rode out Katrina with half a terabyte of data on the line, guarded by a former Green Beret whose preparations included gen
"We're on the 10th and 11th floor of a corporate high rise on Poydras Ave., right near St. Charles. We have generators and tons of food and water. It is five of us total. I am not sure how the Internet connection will be affected. I have a camera and my gun. Sustained winds are 175, gusts to 215. The real danger is not the wind, it's the storm surge the wind will be pushing into the city from the Gulf through the lake. The city might never recover. Honestly, this thing could be biblical"
That online diary, or weblog, entry was posted about noon on Sunday, Aug. 28, by Michael Barnett, a former Green Beret and business consultant to Intercosmos Media Group, the parent company of domain registrar DirectNIC and Zipa.com, a Web host. Barnett, a lifelong friend of Intercosmos chief executive officer Sigmund Solares, was holed up with his girlfriend Crystal Coleman in a data center in New Orleans' Central Business District, an alleged safe place to ride out Hurricane Katrina. Solares hired Barnett as crisis manager for the storm.
Barnett updated his blog during the nation's greatest natural disaster, as wind, rain, floods, looting and death engulfed one of America's most beloved cities. Research firm Economy.com expects economic losses of $125 billion.
According to the U.S. Census Bureau, New Orleans is home to 30,262 businesses, including little-known companies like Intercosmos and well-known ones such as vacuum cleaner manufacturer Oreck Corp. The storm tested both companies' business continuity strategies.
Intercosmos' plan: Back up critical data from its New Orleans data center remotely, but, because of cost, gamble with half a terabyte of Web content from thousands of free hosting accounts bundled with DirectNIC.
Oreck's plan: Move operations from its headquarters and data center in New Orleans to its Long Beach, Miss., manufacturing and call center facility, and vice versa. These operations would be supplemented by a backup data center in Boulder, Colo., and a call center in Denver. Oreck didn't plan on both its New Orleans and Long Beach facilities taking a hit.
The lesson: Plan for a worst-case scenario, but remember that disasters have the potential to exceed expectations.
Here is an account of how these two companies struggled to keep their businesses, people and technology running before and after Katrina blew apart the Gulf Coast. The efforts stress the importance of backing up key data and people in dispersed locations, of being able to strip your company down to its essentials, and of having a staff that can improvise even as the best-laid plans go awry.
Aug. 27, 28: Katrina builds in the Gulf of Mexico and becomes a Category 5 hurricane with winds of 175 mph.
Katrina and Rita didn't just wipe out data centers; they wiped out a lot of disaster plans, too. Check out:
Intercosmos moves some critical functions such as DirectNIC name servers, which are required to keep its clients' domain names functioning, to backup locations in Florida and Atlanta.
CEO Solares, however, keeps half a terabyte of customer and Web content data in New Orleans. That data—supporting the free, basic Web sites customers get with their $15 domain registration—could be restored from backup tapes, if necessary, but at the low prices DirectNIC charges, it can't afford full redundancy. Besides, these customers don't get the same guarantees as if they were paying for their sites through Zipa.com, the company's Web hosting subsidiary.
Still, to protect the company's reputation with its customers, he wants to keep sites online.
Meanwhile, Oreck backs up its last eight hours of activity—invoices, manufacturing requests and shipment information—on tapes to be loaded on its IBM AS/400 mainframes in Boulder. Oreck CEO Tom Oreck, the 54-year-old son of company founder David Oreck, takes the tapes to Houston and sends them to Boulder via Federal Express.
CEO Oreck then launches the company's contingency plan, which transfers the company's call center to Denver, a site that takes call overflow from its Long Beach call center. Oreck employees take all the data and documents—customer data, shipping destinations and inventory information—the company would need to restart later.