The bombing of the World Trade Center and the Pentagon destroyed buildings and lives. It also shattered widely held beliefs.
Even mundane things like power outages now seem sinister. At least 350 buildings housing thousands of businesses in lower Manhattan lost power for several weeks. “When others hear that, it begins to turn their focus from immediate evacuation response to a larger business-continuity plan,” says Alton Penz, president of the Gimbal Group, a contingency planning consultant in Arlington, Va. September 11, he hopes, will make more businesses think, then plan.
Assembling such a plan is no small task. More than 60% of the information technology managers surveyed in 2000 by Gartner said they didn’t believe their companies had a basic continuity plan. But there are places managers can go to for help. Some will assign technical specialists to learn all they can about the topic. Many more will likely hire the services of backup and recovery specialists like IBM, SunGard and Comdisco (which is in the process of being acquired by SunGard). The trio dominates the business with some 75% to 85% of a $2 billion annual market, Gartner says. EDS has a small piece of the market, and there are dozens of smaller disaster-recovery specialists in the U.S.
Whether you subscribe to a business continuity service in advance or activate a recovery plan once disaster strikes, you need a way to regain information lost when the world is turned upside down. Backup tapes, paper files copied for storage offsite, or temporary workspacesknown as “hot sites,” with a carbon copy of the destroyed networkare all options.
“A business-continuity plan is paramount to our business,” says Dan Canzano, chief information officer at payroll-processing giant Paychex in Rochester, N.Y. “One minor catastrophe could put us out of business. September 11 hasn’t changed our thinking. But what it did was confirm in our minds that we have developed the kind of continuity plan here to be able to deal with something as catastrophic and tragic as the twin towers being taken out with thousands killed.”
Paychex does its own disaster planning. When things go wrong, key managers know exactly whom to call. When branches are incapacitated, others take up the slack. And when information systems go down, there’s a backup there, too: Every office’s data are backed up on tape and held offsite, or else mirrored on another branch’s servers. There are no exceptions.
But not everyone wants to handle the complex, tedious job of copying everything a company does. For them there are plenty of contractors to turn to for help. At the most basic level, disaster recovery means cataloging the data a company needs, and then backing it up so that if there’s a business interruption, the company can reload the data that was set aside and continue as before. For some, that means copying information every few days. Larger enterprises usually need to back up at least once a day. Regulations essentially require financial institutions in the U.S. to mirror operations at multiple sites, which explains why many Wall Street companies were prepared for disaster.
Backing up data is hardly the end of the story. A large company must dissect its workflow before it knows which information it really needs. And companies don’t simply have information; they have people who massage it in ways that are never precisely the same.
Those people often write their own software, and that can become a problem if the person who wrote and ran the software becomes incapacitated. Good recovery plans need to take that into account, too. “It’s not a simplistic analysis,” says Ric Hughes, a manager in PricewaterhouseCoopers’ IT architecture practice. “It’s not unusual to take several weeks to look at the interdependencies of applications,” he adds. (For a walk-through on how to analyze the importance of different technology assets, see “Assessing Your IT Exposure,” p. 100.)
The “interdependencies” of people are at least as difficult to square. Beyond mere information, managers must worry about how to care for their people when things go wrong. Coordinating access to employee assistance programs, insurance and even retirement arrangements become more important than ever under those circumstances.
While business-continuity providers can’t deal with every contingency, they can help with basics like providing space and equipment. A contract or subscription with an IBM or a SunGard works like an insurance policy, and in the event of a disaster, the costseven at more than a million dollars a yearcan be worth it.
Of the three major disaster recovery providers, IBM has the most multinational clients. SunGard’s strength lies in dealing with financial services companies. Comdisco has its share of financial services and manufacturing companies, but has been slow in adapting to the shift away from mainframes at many companies, says Julie Giera, an analyst with Giga Information Group.
It was troubled investments in Internet companies that forced Comdisco to file for bankruptcy and put its business continuity unit up for sale. Earlier this month, the company agreed to sell the division to SunGard for $825 million.
In the days following the World Trade Center attack, many companies sought to relocate operations to temporary workspaces in New Jersey and other places near Manhattan. In the chaos, many companies’ disaster plans came up short.
Employees of St. Paul Re, an insurer, safely fled the office building just a block from the Trade Center as soon as the first plane hit. Their contract with SunGard was supposed to give them daily backups, plus spare space in Hoboken, N.J., in case of dislocation. But the contract gave St. Paul Re only conditional dibs on the Hoboken site; others, who had paid more for the privilege, had priority.
In fact, a government agency did need that space. So St. Paul was out in the cold. Demand was so high that SunGard could offer space no closer than Boston or Chicago. St. Paul Re turned to its own offices in Morristown, N.J., 25 miles west of Manhattan, and had operations restored in three days. Many employees worked from home, others doubled up in Morristown. It was a headache, but the company kept operating.
What did Andrew Cole, St. Paul Re’s chief information officer, learn? “We’ll tweak the plan,” he says. “It definitely held up, but we need to get the space dedicated.”
St. Paul Re’s experience shows how disaster plans must constantly be updated. “The important thing is revisiting,” says Al Decker, director of security and privacy services at EDS. “Is the plan current? Is it tested?” If it is not, he says, the next disaster could be your company’s last.
Additional reporting by Matthew Rothenberg and Sean Gallagher