Planning for the Unimaginable

Tom D’Ambrosio is the chief information officer of Multex.com, a New York company that provides information to traders and hosts applications for financial firms.

We have an extensive disaster-recovery plan, but it didn’t take into consideration the World Trade Center collapsing. For that matter, it didn’t take into account both our data centers being down. But that’s exactly what happened on Sept. 11. At 5:45 p.m., the lights went out—just like that.

You have to understand what a loss of power means to Multex. Between our two New York offices, we have about 600 servers. Those servers let us provide instant information to Wall Street traders and let us act as an Application Service Provider to firms such as Merrill Lynch. We link into their portfolio systems or into their marketing systems or other internal systems that they have. We have no other way of doing business. Without the servers, we’re not just down; we’re out.

Even though the data center at our headquarters at 100 William Street is smaller than the primary one at 75 Park Place, we don’t think of it as a backup. We operate the two data centers as if they were one virtual data center, and we have a tremendous amount of dedicated bandwidth between the facilities—more than 8 gigabits per second. We can make the two data centers look like one, by changing the routing and configuration. That became our plan, to run the operation out of the smaller data center at our headquarters, since we had physical access to that building. Our building at 75 Park Place was only a block from the World Trade Center and had been evacuated. We didn’t even know if it was still standing.

Our goal was to be up and running whenever Wall Street and the markets opened. We didn’t know when that was going to be. On Wednesday they said, “Maybe Thursday.” Thursday they said, “Maybe Friday.” By late Thursday, they started saying, “Probably Monday.” We didn’t know. But we needed to be there right with them.

In a situation like this, where your business is at stake, you have to have a response for every contingency—even if you never put it into effect. We had to go on the assumption we wouldn’t be able to get our equipment from 75 Park Place. So we got on the phone to Dell, our main hardware vendor, and ordered hundreds of new servers.

The idea was to bring in new capacity so that the headquarters data center could run the whole business with as little degradation as possible.

As part of a completely different track, our president, Chris Feeney, started arranging to move our entire data center to a third-party site—just in case our headquarters, too, got evacuated. He was talking to vendors we have like MCI Worldcom and Dimension Data. They all said, “Look if you need data center space, we’ve got space. We’ve got space on Long Island, we’ve got space here in the city, space in New Jersey.”

And we had clients, Reuters and others, that called us up and said, “We’ll give you space in our data center if you need a place for your people to sit.”

We don’t have a contract with any disaster-recovery firm. I had talked to companies like SunGard and Comdisco years earlier. But ours is a complicated system; it’s not the kind of system where there is a big switch and you just throw it and you’re switched over. It doesn’t work that way.

We had also looked at a co-location scenario where we would have rented space. But the amount of space we needed, 100 cabinets worth, would have cost us a fortune. Given the economics of the situation, it made more sense to build the second data center, which is what 75 Park Place was, and, thankfully, still is. It’s once again accessible—and is in fine shape. I think the right decision now is for us to build a third data center, potentially in Europe or somewhere geographically isolated within the United States. That was something we were going to look at in the first quarter of next year anyway. We might try to do it a little faster now. — WRITTEN WITH ROBERT HERTZBERG