The atmosphere was tenseBy Edward Cone | Posted 2001-10-29 Email Print
Cantor Fitzgerald and its electronic trading subsidiary eSpeed lost 700 employees, including 180 members of technology staff. The response of those still on the job: Restore service to customers in less than two days....">
The atmosphere was tense, with people not knowing what had happened to their friends or colleagues. "For days, every time a new face came in the door it was an emotional release," says Noviello. "There was a disaster-recovery contact list, but people were seeking to find each other not for work but to find out who was OK."
Noviello would divide most of the next two days between a Manhattan command center set up in borrowed space and his home, using his cell phone, his fiancée's cell phone and a land line. Even so, he frequently patched into the company's London communications system to gain access to numbers he couldn't reach from New York.
Beyond the technical questions were operational details like advising staff on public transportation options to the suburban site, reestablishing shifts and making sure there were counselors on duty. Conference calls every two hours kept track of milestones and objectives. "We were talking at 2 a.m., at 4 a.m.," says Noviello. "Who is sleeping during something like this? Work is great therapy."
On Wednesday, Cantor Chairman and CEO Howard Lutnick told him the bond market would reopen the next morning, and asked if the system would be ready; Noviello spoke to Claus and his lieutenants and came back with a yes.
"We never considered not being there on Thursday," he says. "There is too much dedication and enthusiasm in this group. We said we will be there for ourselves and our friends."
Working in the cold, crowded data center, which normally houses no more than a handful of workers, eSpeed's people relied on their knowledge of the systems and procedures instead of following a written plan.
"We had some major hurdles to cross, but we approached them systematically, and each step worked so there was no need for a plan B," says Noviello. "For days it looked like chaos, but people knew what they needed to do." Senior people worked in shifts, and workers took charge and stepped up as needed. It helped that as a small company, people had had lots of exposure to different systems.
None of this effort would have succeeded without the duplicate architecture in Rochelle Park. Yet Cantor only started moving into the facility in February. The previous disaster-recovery plan was based on a co-location plan at another New Jersey location, but Giaccone, who ran eSpeed's global infrastructure, had pressed for more.
"Joe pushed every day for another facility," says Noviello. "Our ability to work now is a testament not just to the work of these people but to the planning by our former colleague."
From day one, Rochelle Park was seen as a concurrent system, not a disaster-recovery site. The shift was driven by eSpeed's role as the largest player in electronic bond-trading, which meant uninterrupted service was an imperative. The nondescript building in a blue-collar town was perfect—a former telecom facility across from another telecom building. Systems alternated between the trade center and the mirror site, with particular products (e.g., zero coupon bonds) running live for a month at one location and then switching to the other; about half of the company's approximately 40 products were live at each location at any given time. "In that sense we had run our disaster-recovery tests the day before," says Noviello.
The mirror site and the World Trade Center were connected by a high-speed optical line, over which eSpeed linked the storage area networks at each site. Sybase data-replication software mirrored critical databases between the sites. Half of the company's Microsoft Exchange e-mail servers were also located full-time in Rochelle Park.
All that redundancy would be stretched to the limit as eSpeed worked to overcome the technical hurdles between them and the opening of the bond market Thursday morning. Two of those hurdles were huge: the loss of eSpeed's private network connections and the destruction of the company's ability to handle fulfillment of trades.
The critical transaction-processing systems that powered eSpeed's and Cantor's brokerage services had remained intact through the mirror in Rochelle Park. But though the matching engine stayed up and running, all connectivity to the private network had been through the optical connection to the World Trade Center. No one could reach eSpeed's servers from the private network to execute trades.
On Sept. 11, both of eSpeed's redundant private networks serving the company's biggest brokerage customers were still located at the World Trade Center. Plans to connect its network to the public telephone network at Rochelle Park had not yet been finalized. "We've only had the building since February," says Noviello. "It wasn't fully built out, and there were some areas where we wanted to put in finishing touches."
But the Rochelle Park site could connect to the Internet and the London data center. Immediately, Noviello's team started contacting customers to arrange alternative ways for them to connect to eSpeed.
Customers with overseas offices connected to the London data center were rerouted across their own networks to London. ESpeed worked with customers to reconfigure its servers on their premises to point to London, and moved or expanded customers' access to the eSpeed network to connect to that site. For customers without overseas private networks, eSpeed worked to get them access over the Internet until the customers could remap some of their digital connections to the Rochelle Park facility.
The second problem, of fulfilling trades, was more difficult to resolve. Although Noviello's team was able to restore some of the applications that handle the financial back-office functions of the trading system, it was unable to reproduce the settlement system for all of eSpeed's products at the Rochelle Park site. Without being able to clear or settle transactions, eSpeed would be unable to open for business.