Witnessing ChaosBy David F. Carr | Posted 2005-10-04 Email Print
No-Size-Fits-All! An Application-Down Approach for Your Cloud Transformation REGISTER >
Vacuum-cleaner maker Oreck had the linchpin of its business continuity plan blown away by Hurricane Katrina. Intercosmos Media Group rode out Katrina with half a terabyte of data on the line, guarded by a former Green Beret whose preparations included gen
Aug. 31: Looting becomes rampant in New Orleans.
"Right now we're trying to show you all the looting," Barnett reports, providing some narration to go with the video he's broadcasting via Webcam. "Guys pushing shopping carts with 40 Nike boxes in them. People breaking into cars. Assaulting ATM machines. It's hard just to sit by and do nothing."
Barnett manages to reduce the draw on the generator from 30% capacity to 20%. The DirectNIC team answers a call for help from another nearby firm that wants to get access to the data on its servers. The data center team fetches the other company's servers, powers them up and assists with a 40-gigabyte download of corporate data. In return, they get 25 gallons of bottled water and assorted cleaning supplies.
Meanwhile, Oreck's project teams have to plan to start a Long Beach plant with no electricity, no generators and no infrastructure to distribute vacuum cleaners, air purifiers and cleaning products. First, Oreck corrals generators for power and trailers for employees, but still can't print out bar codes, track orders and ship its products without network connections.
To help with logistics, it calls United Parcel Service. Oreck, a longtime customer of UPS for shipping, needs help printing bar codes, picking order tickets and distributing products as if they came from the Long Beach plant.
After consulting with the company's UPS account representative, Richard Behrendt, and other UPS executives, Oreck agrees on a fix. The plan: Allow UPS to tap into Oreck's mainframe systems in Boulder to grab orders and serial numbers and print them out in Atlanta. This information would be attached to products manufactured in Long Beach. UPS would truck in food and water for employees and then leave with Oreck products from UPS' Atlanta facility.
Such an arrangement typically takes six to eight weeks, but Oreck wants to start the Long Beach site by Sept. 9, or five work days. UPS doesn't know Oreck's supply chain system and needs to find people internally who can navigate "green screens" and mainframes. Oreck's mainframe-based supply chain system lacks a manual and customized function keys. Two Oreck employees train the UPS teams on using special commands to hop from one screen, say, pick ticket, to another, such as shipping destination. Labor is taken from other UPS shifts and temporary workers.
Sept. 1: Conditions in the city unravel as police are overwhelmed.
"The coroner's office is shut down so bodies are being covered in leaves at best or left where they lie at worst. Until we get a military presence of significance in the city, the roving gangs of thugs own the streets."
An exhausted New Orleans police officer takes shelter in the DirectNIC offices. The police command and control structure is so shot that the officer didn't know New Orleans is under martial law.
At 9 a.m., Barnett reports that the Federal Emergency Management Agency and the city will permit DirectNIC to get diesel shipped in if someone can deliver it. By 10:30 a.m., a fuel truck is waiting downstairs. The problem? The truck arrives before the data center team can get the fuel drums downstairs. The driver leaves.
At this juncture, DirectNIC 's high-speed Internet connections are failing as backup power for telecommunications substations around the city fades, making it necessary for DirectNIC to ditch some Web sites and services. One casualty: Something Awful, a humor Web site, which is hosted at rock-bottom rates and consumes too much bandwidth. When he took on the account, Solares cut a deal that allows for the site to be dropped in a crisis.
Outside, the situation worsens. Dead bodies in the streets and disorganized relief efforts prompt Barnett to send a distress signal: "In case anyone in national security is reading this, get the word to President Bush that we need the military in here NOW. The city is being lost .... We need the kind of logistical support and infrastructure only the Active Duty military can provide."
That afternoon, the diesel delivery succeeds on the second try. Solares tries to contact his company's telecommunications providers, BellSouth and TelCove, to revive connections. Meanwhile, Barnett and Solares wade through filthy water to supply City Hall with an Internet router for emergency communications.
That night, Solares tries to contact employees now working remotely to say he plans to pay them whatever they earned in their last pay period, whether they've worked or not. However, Solares can't reach the accounting staffers responsible for payroll.
For Oreck, the next four days are spent procuring trailers for homeless employees, lining up generators for its Long Beach plant and finding its workers. CEO Oreck commutes between his Houston hotel room and the company's temporary facility in Dallas.
Sept. 2: Military arrives in New Orleans to bring aid and restore order.
"This place is completely coming apart. The hopelessness on the street breaks the heart. The old, the tired, the sick seem resigned to their presumed fate. Death."
Looking down on New Orleans from the roof, 27 stories above the ground, Barnett senses the city will never be the same. He also doesn't like the way "stressed-out, trigger-ready police and military types" look at him while he's waiting for a fuel delivery.
Noah Lieske, a co-owner of Intercosmos, adds a blog post requesting a place for 55 people to work temporarily, ideally with 6,000 to 8,000 square feet of office space and a data center nearby.
Meanwhile, Oreck is on the move. With Oreck's New Orleans location inoperable for an undetermined time, executives and workers representing technology, payroll and human resources populate a 20,000-square-foot IBM disaster recovery site in Dallas with 100 workstations. CEO Tom Oreck, a longtime IBM customer, procured the space after the levees broke in New Orleans, but hopes the company's stay doesn't last too long.
"We're fully functional there, but it is close quarters and there's only so long we can do that," Oreck says. "If we can get back to our facility in the next 60 to 90 days, we'll be OK. If longer, we'll have to find other arrangements in Dallas or Houston—temporary sites for the longer term." 8