Salesforce.com: When On-Demand Goes Off - ' The Big Fix '

The Big Fix

Well before the outages occurred, Salesforce.com announced it was spending $50 million on Mirrorforce, the company's effort to expand its data centers to provide real-time fail-over protection to customers. "We were investing in redesigning our entire infrastructure," says Kendall Collins, vice president of product marketing for Salesforce.com.

This was an extensive undertaking for the company. "We had to rebuild our hardware infrastructure to support [Mirrorforce], and we have rewritten every piece of Salesforce.com software, from the application to integration, caching, database search and database management," Benioff said regarding the ambitious initiative, which was rolled out during the outages in late 2005 and early 2006.

Previously, Salesforce.com had been serving its entire customer base from a single, third-party Web hosting facility in Silicon Valley, operated by Equinix, the global provider of network-neutral data centers; the company used SunGard Data Systems, the software and processing solutions provider, for access to a remote disaster recovery site. This system had been providing an uptime rate in excess of 99%, the company maintained. Salesforce.com's accelerated growth and the switch to new data centers, however, overextended the system's capabilities and contributed to the outages, according to company spokespeople.

The key element in Mirrorforce is a mirroring system that creates a duplicate database in a separate location and synchronizes the data instantaneously. In the event that one database is destroyed or disabled, the other takes over. Much of this function was built in-house.

Specifically, Salesforce.com built new data centers on the East and West coasts, while retaining its Silicon Valley facility. Plus, it built an additional West Coast facility to support new product development. In addition, Bois says, the company has spread out its larger customers to balance its database load.

By March 2006, Salesforce.com seemingly had beefed up its information-technology infrastructure to the point where the outages were no longer occurring. The postings on gripeforce.blogspot.com dried up. "The service drastically improved," says ASCAP's Johnson. "We've virtually had no problems."

"The service has been as close to perfect as possible," adds Wesley Benwick, CEO of Bennett's Business Systems in Jacksonville, Fla., which sells copiers and digital imaging services.

"The vast majority of our customers were not concerned and had faith in us that we were going to power through and fix this and get back to our historic high levels of reliability and availability, which we did in a very short period of time," Francis adds.

Next: ' Providing Transparency ' >>

>>> More Projects: Customer Service Articles          >>> More By Baselinemag