Disasters of all kinds present the same problem: They are sudden, and their effects are spread unevenly.
When it comes to staffing, you probably won’t suffer a sudden loss of 10% of your information technology personnel.
Instead, you’ll likely lose 100% of some key department, the equivalent of one-tenth of your IT operation. Or you’ll lose just a few key people. But if those few key people alone know the details of how your servers and networks work, they can’t be replaced under any circumstance.
In fact, loss of a few key individuals could be more crippling long-term to a company than losing a larger number of people. “It’s the key knowledge that matters,” says Jon Scarpelli, a vice president at Ciber, the computer systems hosting company. “It wouldn’t matter if I had 50 operators. If I didn’t have the key people who know the data and process flows, I’d be up a creek.”
The key to backing up staffers is not just more people. It’s codifying what staffers know. Far too often, companies don’t consider that they might have to turn to outside help in a crisis. Those helpers will come in cold and be asked to operate critical applications and systems.
Documentation is the only solution. Companies must define who knows whatand document it, says Mike Symmers, a senior manager with Accenture in Chicago. “I wish I had the answer to capturing everything in people’s heads. There’s no online brain dump. Having said that, we do need to document what we know.”
It’s not enough to document only what your current employees know at the moment about existing systems and hardware. All new work, all changes, must be documented as well. It takes a “military mindset,” says Jon William Toiga, a disaster recovery consultant.
New focus on disaster readiness like that could mean renewed interest in knowledge-management techniques. “When people die, retire, get sick, they walk away with 80% of what they know,” says Stuart Robbins, founder and executive director of the CIO Collective and CEO of knowledge management vendor Kmera. “There are already (knowledge management) tools and methodologies out there that can help with this.”
But in many cases, backing up people needn’t take a great deal of science. One important, seemingly mundane task is to make sure a company has a current, complete organizational chart that specifies who reports to whom and what, precisely, they do. Then, if someone is suddenly not available, it’s clear what tasks will go undone unless that person is replaced.
The most logical source of replacement talent is internal, but that’s only feasible if you’ve thought ahead. Smart companies will constantly cross-train information technology employees so that they become switch-hitters.
Or companies will run the same tasks in two or more shifts at the same site or in simultaneous shifts at two different locales. That way, there are always two platoons of workers who know the other’s tasks. In a minor disaster, they can take up the additional work and keep data flowing.
“With most data centers having alternating shifts of workers, someone who knows the processes is going to survive,” says Mark Stonecipher, associate partner at Accenture. “If you only train one shift, you’re in deep yogurt.”
Long before the earthquake that disrupted the 1989 World Series between San Francisco and Oakland, the Bank of America had set up twin data centers in San Francisco and suburban Concord. “The main question around relocating staff to Concord was how to have staff available after an earthquake,” says information technology consultant Max Hopper.
For BofA, disaster planners concluded they could ensure one or two levels of backup as long as they had adequate staff in the data centers. Top executives could remain based in San Franciscobut move to Concord to work if necessary.
Backup personnel can also come from outside sourcesdisaster recovery services and systems integrators. Comdisco, for instance, has about 200 consultants who will go to any disaster area and help replace a company’s lost employees until the business is back up and running. For extra help, Comdisco calls on five or six smaller companies that know Comdisco’s methodology and can help when needed, says Damian Walch, senior vice president of professional services.
Electronic Data Systems (EDS) and Computer Sciences Corp. (CSC) also operate huge data centers worldwide and provide backup data services, business continuity planning and disaster recovery to customers. Consulting giants such as Deloitte & Touche, Accenture and KPMG also help clients plan for disaster and business recovery.
About 1,000 EDS employees helped companies in the World Trade Center recover, and moved more than 4,000 PCs and other equipment into New York. In addition, they had 61 people at the Pentagon, where teams helped restore voice and data connectivity for Navy workers.
But most disaster providers don’t deal with staffing and they can’t replace your people, says Toiga, the independent disaster recovery consultant. “They do hardware and network replacement,” he says.