Planning for the Unimaginable

Tom D’Ambrosio is the chief information officer of Multex.com, a New York company that provides information to traders and hosts applications for financial firms.

We have an extensive disaster-recovery plan, but it didn’t take into consideration the World Trade Center collapsing. For that matter, it didn’t take into account both our data centers being down. But that’s exactly what happened on Sept. 11. At 5:45 p.m., the lights went out—just like that.

You have to understand what a loss of power means to Multex. Between our two New York offices, we have about 600 servers. Those servers let us provide instant information to Wall Street traders and let us act as an Application Service Provider to firms such as Merrill Lynch. We link into their portfolio systems or into their marketing systems or other internal systems that they have. We have no other way of doing business. Without the servers, we’re not just down; we’re out.

Even though the data center at our headquarters at 100 William Street is smaller than the primary one at 75 Park Place, we don’t think of it as a backup. We operate the two data centers as if they were one virtual data center, and we have a tremendous amount of dedicated bandwidth between the facilities—more than 8 gigabits per second. We can make the two data centers look like one, by changing the routing and configuration. That became our plan, to run the operation out of the smaller data center at our headquarters, since we had physical access to that building. Our building at 75 Park Place was only a block from the World Trade Center and had been evacuated. We didn’t even know if it was still standing.

Our goal was to be up and running whenever Wall Street and the markets opened. We didn’t know when that was going to be. On Wednesday they said, “Maybe Thursday.” Thursday they said, “Maybe Friday.” By late Thursday, they started saying, “Probably Monday.” We didn’t know. But we needed to be there right with them.

In a situation like this, where your business is at stake, you have to have a response for every contingency—even if you never put it into effect. We had to go on the assumption we wouldn’t be able to get our equipment from 75 Park Place. So we got on the phone to Dell, our main hardware vendor, and ordered hundreds of new servers.

The idea was to bring in new capacity so that the headquarters data center could run the whole business with as little degradation as possible.

As part of a completely different track, our president, Chris Feeney, started arranging to move our entire data center to a third-party site—just in case our headquarters, too, got evacuated. He was talking to vendors we have like MCI Worldcom and Dimension Data. They all said, “Look if you need data center space, we’ve got space. We’ve got space on Long Island, we’ve got space here in the city, space in New Jersey.”

And we had clients, Reuters and others, that called us up and said, “We’ll give you space in our data center if you need a place for your people to sit.”

We don’t have a contract with any disaster-recovery firm. I had talked to companies like SunGard and Comdisco years earlier. But ours is a complicated system; it’s not the kind of system where there is a big switch and you just throw it and you’re switched over. It doesn’t work that way.

We had also looked at a co-location scenario where we would have rented space. But the amount of space we needed, 100 cabinets worth, would have cost us a fortune. Given the economics of the situation, it made more sense to build the second data center, which is what 75 Park Place was, and, thankfully, still is. It’s once again accessible—and is in fine shape. I think the right decision now is for us to build a third data center, potentially in Europe or somewhere geographically isolated within the United States. That was something we were going to look at in the first quarter of next year anyway. We might try to do it a little faster now. — WRITTEN WITH ROBERT HERTZBERG

Feeling stuck in self-doubt?

Stop trying to fix yourself and start embracing who you are. Join the free 7-day self-discovery challenge and learn how to transform negative emotions into personal growth.

Join Free Now

Picture of Tom D'Ambrosio

Tom D'Ambrosio

TRENDING AROUND THE WEB

If you tend to worry too much about things you can’t control, say goodbye to these 8 behaviors

If you tend to worry too much about things you can’t control, say goodbye to these 8 behaviors

The Blog Herald

8 subtle signs a person has low-quality energy, says psychology

8 subtle signs a person has low-quality energy, says psychology

Global English Editing

If you recognize these 7 subtle signs, you’re in serious need of a digital detox

If you recognize these 7 subtle signs, you’re in serious need of a digital detox

Personal Branding Blog

People who genuinely enjoy life despite not being rich usually display these 7 unique traits

People who genuinely enjoy life despite not being rich usually display these 7 unique traits

Small Business Bonfire

People who are secretly unfulfilled in life usually display these 7 behaviors (without realizing it)

People who are secretly unfulfilled in life usually display these 7 behaviors (without realizing it)

The Vessel

People who stay happy and active into their 70s and beyond usually have said goodbye to these 10 habits

People who stay happy and active into their 70s and beyond usually have said goodbye to these 10 habits

Jeanette Brown