Salesforce.com: When On-Demand Goes Off

A little more than 12 months ago, Salesforce.com, the Web-based, on-demand customer relationship management (CRM) and business services provider, was hit by a series of service outages over a six-week period that affected as many as 350,000 subscribers. The incidents jeopardized Salesforce.com’s relationship with some of its own customers and potentially sidetracked efforts to extend the vendor’s reach into large enterprises.

Fast-forward to mid-November 2006, when the software as a service (SaaS) company announced its third-quarter results for fiscal 2007. Revenue came to $130 million, a 57% jump over the same period in fiscal 2006; net paying subscribers were up 61,000, to 556,000; net customers up 2,300, to 27,000. That’s the kind of growth that could give SaaS a good name.

Yet, the service outages—which have since been resolved— raise an issue for chief information officers to consider if they are looking to use on-demand software to deliver applications—whether it’s from Salesforce.com or any other vendor.

Salesforce.com’s problems began on Dec. 20, 2005, when its clients could not access the company’s servers—and obtain customer records—for more than three hours. The problem: a rare database software bug, said the company’s CEO, Marc Benioff. Though Salesforce.com uses Oracle database software—Oracle Database 10g and Oracle Grid Computing–as key technology enablers for its service and internal operations, Benioff declined to assign any blame for the bug. However, Bruce Francis, the company’s vice president of corporate strategy, said at the time, “The vendor [Oracle] has one of the largest and most sophisticated development teams on the planet, and they had never encountered this bug before. Salesforce.com is working with [them] … to make sure that the bug doesn’t crop up again.” The bug hasn’t reoccurred since, he said.

Salesforce.com’s vulnerabilities, however, extended beyond the bug. The company experienced two more outages in January and several in early February. These were caused by “system performance issues,” the company said. Typically, such problems varied. On Feb. 9, 2005, for example, a primary hardware server failed and one of the company’s four North American servers, NA1, did not automatically recover. “This required a manual restart of the NA1 database,” Salesforce.com told customers on its Web site. The outage lasted a little more than an hour.

Earlier, a Jan. 30 performance problem was attributed to a shortcoming in the company’s database cluster, a collection of databases. This issue required Salesforce.com to restart each database instance in the cluster, resulting in a pause in service. The system was down for about four hours, and even when the company brought the service back up, the application programming interface (API) remained disabled for several more hours.

Clearly, Salesforce.com had some infrastructure issues, and with them a growing credibility gap. Its challenge was to move quickly and decisively in making fixes and enhancements, beginning with its database and service capabilities.

The outages came at the worst possible time for Salesforce.com, which was growing rapidly–its annual revenue of $309.9 million for the year ended Jan. 31, 2006, represented an increase of 76% over the prior year. It was also seeking to attract more large enterprise customers with new software and services offerings. “They’re not just a CRM company any longer, but a business services provider,” says Rob Bois, a senior research analyst with AMR Research in Boston.

Salesforce.com intended to drive this change with its January 2006 launch of AppExchange, an online marketplace for applications and Web services from other vendors and developers that could be customized and integrated with Salesforce.com’s core CRM service.

Salesforce.com was also gearing up to compete more aggressively with traditional enterprise platforms. With enterprise customers concerned about trusting reliability and availability to an outsider under the SaaS model, the outage problems weren’t exactly confidence builders for the targeted audience. Moreover, by its own admission, management failed to communicate problems to customers after the initial outages. It learned from its mistake, however, and soon shored up its customer communications processes—a positive move in most businesses and an essential component in the CRM on-demand arena, where unexpected downtime can cost customers millions in lost sales.