Symantec`s Midnight at the Oasis

By Lawrence Walsh  |  Posted 2008-03-31

Ordering Symantec software is a breeze for Stephen Nacci, director of vendor relations at TLIC Worldwide, a software reseller in Rhode Island.

Processing product and licensing requests through the vendor’s online portal has never been easier, and the amount of attention the reseller gets from his regional representative has never been higher. Nacci’s company has even gotten calls inquiring about the quality of customer service and support from Symantec CEO John Thompson.

“They’ve got it together a lot better,” says Nacci. “We’re getting a lot more timely responses from them—quicker service, more attention and better responses to our needs.”

Doing business with Symantec today is a far cry better than it was just one year ago at the height of a crisis that shook the security and storage software vendor to its core.

In November 2006, Symantec flicked the switch on a new enterprise resource planning (ERP) system designed to let partners and customers order products and activate licenses. Internally, it was called Project Oasis, a major step in the merger of Symantec and Veritas. Once implemented, the combined company’s vast network of resellers, integrators, distributors and customers would place orders through a single system that included all the tools and information needed for intelligent software purchasing.

Technically, Project Oasis, an upgrade to Oracle 11d, was flawless. The code, interface and system—aside from some conflicting records that made accounting difficult to interpret—went exactly as Symantec had conceived when it launched the project in May 2005.

But users didn’t understand the system. The voluminous information it provided them and the myriad steps required to place orders created confusion and poor usability. In fact, the system was so cumbersome that resellers and customers were on the verge of revolt. The problems were so acute that CEO Thompson blamed part of the company’s sluggish financial performance in the third quarter of fiscal year 2007 on the ERP plague.

“It probably cost them business,” Nacci recalls. “The system made it so you didn’t want to get involved with them.”

Despite all its planning, Symantec made numerous and near-fatal errors in preparing its users for the change in its ordering system. For one thing, the project management team failed to coordinate the rollout with other divisions that were launching new products at the same time. Despite its best efforts, the management team could not adequately prepare its more than 60,000 resellers, partners and distributors—and scores more customers—for the procedural changes required by the new system.

From the ashes of Project Oasis rose Project Nero, named for the Roman emperor who, the anachronistic legend has it, torched his capital city while cavalierly playing his fiddle. Symantec insiders say the name was chosen to convey the urgency and seriousness of the situation. If the problems weren’t resolved quickly, it could spell financial ruin for the vendor.

*Some companies choose to stay away from ERP systems altogether. Take  Fortune 500 Wesco International with $5.3 billion in revenues.

Project Nero, an ongoing effort, did more than just resolve the problems associated with the ERP implementation. Like the failed project it was designed to correct, Project Nero set off its own chain of events and initiatives that have transformed the way Symantec views its customers and runs its business. After a painful and arduous recovery process, Symantec has not only recovered, but now serves as an example of how to interact with customers and partners in the Web-enabled marketplace.

An Upgrade of Necessity

If the $13.5 billion merger of Symantec and Veritas in 2004 was a marriage of equals, Project Oasis was the wedding band and icing on the cake. To understand the complexity of the task of unifying the two companies under one ordering and licensing system, you first have to understand that this merger wasn’t textbook.

Synergies that theoretically result in cost reductions and savings didn’t exist in this merger. Symantec and Veritas were more than just two different technology providers; they were distinct organizations with different pedigrees. Symantec grew up in the consumer market and sold that way, even with its enterprise business. Veritas was always an enterprise software vendor, and its sales, product delivery and marketing infrastructure was geared toward large-site licensing.

Project Oasis would unite those disparate business models and provide a home for the more than 250,000 products in the combined company’s stable. Oasis would be used to unify Symantec and Veritas ordering and financial systems, launch a new licensing model and license-key release system, establish compliance with global trade regulations and provide new tools for partner enablement. In addition to Veritas’ assets, planners also had to consider the dozens of smaller vendors Symantec had gobbled up in the past five years.

By the time David Thompson (no relation to the CEO) joined Symantec as chief information officer (CIO) in early 2006, Project Oasis was nearly a year into its planning and integration. Thompson’s predecessors had planned to roll out the ERP system in November 2006 in one massive push. However, Thompson, a former Oracle IT executive and veteran of many ERP implementations, recast the project in four major phases, the last being the full ERP launch and the closing down of the old system.

Despite program adjustments, the project never slipped off its intended launch date. In fact, the project launched a week early.

The system was a significant change from the previous iteration. Based on feedback from partners and customers, Symantec designed the system to show copious amounts of information, including every license previously purchased and currently supported, as well as a vast array of parts numbers and product stock-keeping units.

*ERP systems are meant to bring centralized views in to disparate systems. International Game Technology, a multi-billion dollar maker of slot machines, did just that. Check out the article: ERP: Gaming Company Hits Jackpot.

While not the fabled single-pane-of-glass reporting tool, Oasis would give users almost everything they’d need to gain deeper insight into existing licenses and make better buying decisions.

Theory always works better in the lab than in the wild, as Symantec was soon to discover. For users, ordering new software suddenly became cumbersome. The automated system required numerous steps none of the users had performed previously. The system didn’t necessarily leave users hanging, but the additional steps gave users the sense they weren’t placing orders correctly. The resulting frustration and confusion had customers swamping the Symantec support call center, resulting in hold times of 30 minutes or more.

“When they went live, it was one of the most painful processes,” recalls Jodi Honore, vice president of vendor management for software at Ingram Micro, one of Symantec’s largest distribution partners.

It sounds like mayhem, but Symantec management describes it as watching storm clouds gather in the distance. From the moment the system went live, systemic problems and baffled partners smoldered like embers in a lightning-sparked brush fire. Within a month, the fire was raging. By April 2007, it was an inferno.

Ironically, the system was performing as designed—to a fault. The problem was that Symantec had inadvertently created the perfect storm: It had failed to consider the user experience with the new system, it hadn’t correctly identified the true users of the system and it had layered other projects on top of the ERP implementation, thereby complicating the launch and generating more confusion.

*Some companies choose to stay away from ERP systems altogether. Take  Fortune 500 Wesco International with $5.3 billion in revenues.

Compounding the situation was the simultaneous launch of a new, integrated licensing program that unified Symantec’s and Veritas’ schema. Partners and distributors were enraged by the new system. Implementers couldn’t make sense of the information Symantec’s portal was giving them. Competitors salivated at the opportunity to steal market share amid the chaos.

“The amount of change was just too great,” says Julie Parrish, vice president of Symantec’s Global Channel Office. “The human brain cannot possibly absorb all the changes we threw out to them.”

The extended Symantec ecosystem was near a full-blown revolt. If the problems weren’t fixed, Symantec could lose serious revenue and market share. Project Nero was launched to diagnose the problems, correct the fielding of Oasis, and ensure that systems and processes were put in place to avert a repeat of the disaster.

So You’re the Customer?

Project Oasis wasn’t designed in a vacuum. Symantec built the system with the advice and counsel of scores of customers and partners. Buyers told Project Oasis planners they wanted more purchasing histories and licensing analytics, as well as a better system for mixing and matching the tens of thousands of applications, developer kits and APIs.

What emerged from the ashes of Project Oasis was something Symantec didn’t expect: a community of customers it hadn’t known existed and hadn’t previously touched. Behind all the purchasing agents and account representatives that placed orders were armies of implementers, administrators and managers. These people knew nothing about the purchasing process, nor did they care. They wanted to be able to get the licenses they needed to complete their projects without hassle, delay or copious amounts of paperwork.

Suddenly, Symantec was dealing with people it had never fully considered—internal and external users, partners and resellers, customers’ in-house consultants, administrators and integrators. “The contact points in a customer just exploded,” CIO Thompson says.

Symantec was facing a whole new constituency: people with whom it had had little previous contact. The changes in the ordering and licensing system disrupted the processes these people were accustomed to, and they weren’t happy about it. Symantec discovered that these administrators and implementers were more than just vocal: They were influential in the purchasing cycle. Because they were the ones who actually handled product—post-sale deployments and maintenance—their experiences had tremendous influence over future purchasing decisions.

“The user base that was moved to this new model—what I call back-office IT-type resources who are heads-down technologists who apply the technology—was being forced to interact with Symantec in this licensing process, and they didn’t like it,” Thompson explains. “They’re a very vocal community.

*ERP systems are meant to bring centralized views in to disparate systems. International Game Technology, a multi-billion dollar maker of slot machines, did just that. Check out the article: ERP: Gaming Company Hits Jackpot.

“Some saw value from the change—the ones who did requested it. Others found it to be an extra step that they didn’t want to take. So there’s a balance here: It’s change management in the business process.”

The system wasn’t broken, but the way Symantec dealt with its constituency was. While Project Oasis was being developed and deployed, the rest of Symantec was moving forward with other initiatives that the ERP team discovered only in the aftermath of the rollout.

As Oasis was hitting the data center, the new version of Symantec Backup Exec 10d (a venerable Veritas product and one of the company’s overall best sellers) was being launched. Additionally, the change in the licensing schema was done independently of Oasis and, in hindsight, should have been included with the ERP project. Lastly, partners and customers were fuming over the mainstay of Symantec’s security business—Symantec Enterprise Antivirus 10—which many described as a resource hog that dragged down client performance.

Cleaning Up the Mess

Jeff Russakow stepped into the fray at the height of the crisis, taking over the Global Enterprise Support Services division in April 2007. While CIO Thompson worked on resolving technical kinks and refining the user experience, and Parrish soothed irritated partners, Russakow raced to get customers off the phones and smooth out the ordering process.

The mess was more than Russakow could imagine. Partner and customer hold times for support skyrocketed from an average of 2 minutes prior to Project Oasis to an average of 25 minutes. Once a customer got a human on the phone, they could spend up to 20 minutes working through the new process. While Symantec did increase call center capacity prior to the launch of Project Oasis, it vastly underestimated the volume of problems users would encounter.

Russakow rushed 170 “customer care” representatives through training and into the call center to increase capacity. The arrival of the cavalry reduced wait times to more tolerable levels. The nature of the calls and the troubleshooting his teams were doing provided valuable intelligence to the steering committee that was working to resolve the ERP issues. It was through the call center that Symantec discovered the folly of its system design. One of the biggest problems was dirty data.

Symantec had designed the system to show customers their existing licenses. What project planners hadn’t accounted for was that one company could have multiple accounts, because branch offices and divisions often bought licenses separately from the central office, and many companies had multiple accounts with variations of their corporate names. This created conflicts and incomplete data sets in the reporting system, which exacerbated the confusion.

Symantec partners and customers say the company’s customer service responsiveness has improved tremendously in the wake of Project Oasis. One of the key lessons Symantec learned from the debacle was that service still matters, especially to customers and constituents that rely on a large IT vendors for expertise and support.

“More and more, service is a differentiator, especially in the midmarket and among SMBs [small and medium businesses] that have a small IT department,” Russakow says. “They’re looking at the level of service to differentiate who to buy from. It’s all about doing more for the customer rather than differentiating on the product.”

*Some companies choose to stay away from ERP systems altogether. Take  Fortune 500 Wesco International with $5.3 billion in revenues.

Eliminating confusion and improving the product ordering process wasn’t enough. Symantec realized that it needed to recapture the hearts and minds of its customers and partners. Project Nero, the follow-up to Oasis, established several strategic objectives to improve everything from Symantec’s change-management processes to continuously measuring customer loyalty.

Project Nero’s starting point was damage control. Symantec’s executive team launched a concerted campaign to personally reach as many partners and customers as it could to assure them of the company’s commitment to ensuring their satisfaction and earning their loyalty. For months, CEO John Thompson, CIO David Thompson, Chief Operating Officer Enrique Salem and Vice President Julie Parrish—along with other executives—crisscrossed the country and spent countless hours talking to customers and partners about their experiences with Symantec.

Each meeting and phone call generated feedback and action items for the technical and customer service teams. The intelligence collected helped Symantec craft a new corporate-wide strategy for interdivision communications and coordinated product launches. With each step of improvement, Symantec measured the effect on customer loyalty.

“There’s always a lot of learning,” says Salem, who joined Symantec after its acquisition of his company, Brightmail, in 2004. “There were many things we fixed. What you’ve got to get good at is making sure that you focus on improving the experience for everyone involved—the customers and the partners.”

Customer Loyalty 

Project Nero didn’t introduce the concept of customer satisfaction and loyalty to Symantec. In 2004, prior to the Veritas acquisition, CEO Thompson and former COO Jonathan Schwartz developed an initiative to measure customer loyalty, whereby the company would continually survey customers to weigh their satisfaction and their likelihood to remain customers. The instrument of choice: Net Promoter, a popular methodology for measuring customer loyalty.

Aisling Hassell, vice president of customer experience and online, runs the program. She had a front-row seat at the Project Oasis meltdown, monitoring the loyalty scores as they sank to historic lows compared with internal and industry benchmarks. Just as Russakow was providing tactical intelligence on problems customers were experiencing in the field, Hassell’s group was supplying insight on the effectiveness of improvements, along with strategic guidance for prioritizing ongoing improvements and new action items.

As Hassell describes it, her group provided Symantec with a “North Star” to follow. “What’s challenging is making a system that keeps the customer in the forefront and building a way to keep the customer voice loud and clear,” Hassell explains. “A year on, the customer voice is loud and clear with us.”

Measuring customer loyalty is no passing fancy at Symantec. The executive leadership committee—which is chaired by Salem and includes division and department vice presidents, as well as CIO Thompson and Parrish—meets every two weeks to review loyalty metrics and identify what each unit manager is doing to improve their scores and efficiency. “It’s really changed the culture and gotten a lot of things fixed,” Hassell says.

Technology may be the engine that drives business, but business is still conducted by people. Understanding the needs, desires and experience of the customer—whether that person is an internal user, a reseller partner or a consumer—is critical to any company’s success and growth.

“Early on, it was all about speeds and feeds, feature wars and functionality,” Salem says. “But the industry has matured to the point at which you now work on many intangibles. How do people feel when they work with your company? How do we do the right things for our customers?”

While the company won’t release its Net Promoter survey results, Symantec reports that loyalty scores among business and reseller partners have improved eightfold in the past year and are now on par with industry benchmarks.

As a result of Symantec’s use of Net Promoter and the organizational postmortem conducted under Project Nero, the vendor has a master product release calendar, formalized interdepartmental lines of communications for IT project implementations and an improved change management process—all designed to ensure there are no slips in the customer experience.

“You think you should focus on the things that drive revenue, but you forget the importance of experience,” Parrish says. “If you believe the methodology, the experience does drive revenue.”