Disaster Recovery: Virtual Project Yields Tangible Benefits
Champion supplies oil and gas companies with a variety of chemicals used in their operations, such as corrosion inhibitors for pipelines and fluids used in drilling. With oil prices remaining near record highs, Champion has seen demand for its chemicals steadily increase in drilling hot spots around the globe.
The privately owned Houston company has recently been growing at a rate of 20% per year, with revenue of about $600 million in 2006. It is currently on pace to top $1 billion in revenue within four years. That growth, however, was severely straining the company's information-technology infrastructure.
Etcheverry discovered soon after coming on board that Champion's SAP enterprise resource planning (ERP) platform, which was housed on an IBM AS/400 computer, was in desperate need of an upgrade. Reliability and performance were deteriorating as more users were added to the system. Outages were becoming a weekly nuisance, and the AS/400's 350-gigabyte storage capacity was almost maxed out.
What's more, the company was unable to extend SAP to its regional headquarters in Canada, Russia, Scotland, the Middle East, Singapore and Argentina because of networking issues. The company was using a virtual private network over the Internet to connect its regional offices, but the network was too slow to support SAP. SAP requires a low-latency network to function properlythat is, it requires a network that can pass packets of data quickly.
"Our goal is to be No. 1 in this business," Etcheverry says, "but our systems weren't up to the challenge."
Over the past two years, Etcheverry and his team at Champion set about to fix those problems. Their solution, which included implementing a virtual storage system, yielded a number of unexpected benefits, including a much more streamlined and reliable disaster-recovery system.
The first challenge the technology department tackled was its network issues. As Etcheverry says, "We tried to focus our people on the things that would have the most impact on our day-to-day business." In total, Champion employs about 43 people in its technology department and operates on an annual I.T. budget of about $12 million.
Champion contracted with Verizon Communications to implement a Multiprotocol Label Switching (MPLS) network that takes advantage of nodes Verizon has installed in 121 countries. MPLS is a widely supported method for speeding up data communications over IP networks, utilizing a packet labeling system to steer traffic. In so doing, it provides benefits such as allowing administrators to place higher priorities on certain types of traffic such as voice or video. In Champion's case, it was able to leverage the new MPLS network to implement voice-over-Internet Protocol (VoIP) phone service.
Next, Champion turned its attention to upgrading its core computing platform. The company's SAP software was housed on an AS/400 IBM minicomputer (IBM renamed the AS/400 System i in 2006) with 360 GB of built-in storage. The AS/400 was three years old at the timehardly a relic by minicomputer standardsbut Champion had already outgrown its capabilities.
Etcheverry, working with consulting partner RealTech, a German-based SAP specialist, first considered installing a newer, more powerful AS/400, at a cost of about $485,000. But Etcheverry wasn't keen to remain on the AS/400. For starters, he found it difficult to hire quality AS/400 programmers. Recruiting searches had proved expensive and didn't produce the quality of candidates he expected. His preference instead was to convert over to a Wintel (Windows/Intel) platform, which would offer him a wider range of hardware and software choices, as well as a much bigger talent pool.
On the surface, switching to a Wintel platform seems immensely more complicated than staying with the AS/400. With the minicomputer, all elements of the SAP system are housed in the one boxthe SAP client, the DB2 database that serves SAP, as well as the 360 GB of storage. It's a nice, tidy package. The Wintel alternative, by comparison, is the equivalent of replacing a bus with a fleet of taxicabs.
Champion's SAP environment, which includes servers for applications, a "sandbox" for playing with new configurations, a development client where development activities take place, and a quality client where configuration changes are tested, now consists of 11 Dell 2850 two-core-processor servers. Each server costs about $6,500. The database system was converted from DB2 to Microsoft SQL, and is housed on two Dell 6850 four-core-processor servers, running about $13,000 each. The final component, the storage system, is housed on a series of RAID (redundant array of independent disks) boxes from Dot Hill Systems of Carlsbad, Calif., providing a combined total of 8.5 terabytes (TB) of storage. Champion also has an additional 7 TB of storage from Dot Hill for secondary systems such as its Microsoft Exchange e-mail system.
"One might argue that an AS/400 environment is much simpler to manage, and to a certain extent I would agree," Etcheverry notes. "But we went down this path because I couldn't find the people we needed, and I already had people trained in Microsoft on staff. Also, I didn't like being held to one platform and proprietary software. Now I have more options and more bargaining power."
Early in the project evaluation process, the tech team looked at implementing a virtual storage system. Etcheverry already had some experience implementing a virtual SAN (storage area network) at his previous employer, Cameron International, another Houston oil and gas specialty firm, and had gained an appreciation for the cost and ease of expansion benefits it provided.
A further consideration was that Champion needed a better disaster-recovery plan. The company had a bare-bones arrangement in place utilizing a SunGard Data Systems facility in Houston. Under the terms of the agreement, SunGard would make an AS/400 available in an emergency, and had two weeks to essentially duplicate Champion's SAP environment, utilizing data that was backed up regularly. But when Hurricane Rita struck the Gulf Coast in September 2005, the danger of having the company's backup facilities in the same city as its headquarters became quickly apparent. So did the inadequacy of the arrangement. Two weeks was far too long to be without the company's core enterprise system in the event of a real emergency.
"Our old [disaster recovery] system was outdated, and we had strong doubts whether we could continue to run our environment seamlessly if there were a disaster," Etcheverry says. "We didn't ever want to be put in that position again."
Rita spared Houston, but to this day Etcheverry wonders what would have happened if Houston had taken a direct hit.
The Virtual Plan
Etcheverry selected FalconStor, a data storage and protection systems vendor in Melville, N.Y., to implement the virtual SAN system and disaster-recovery system. He had previously worked with FalconStor at Cameron, and was confident the vendor would meet Champion's needs. Beyond that, Etcheverry says he often prefers to work with midsize technology vendors because he can form a closer partnership than with, perhaps, an EMC or IBM. FalconStor had $55 million in revenue in 2006.
The heart of the system is the FalconStor IPStor server. It acts as a form of orchestrator over the entire storage network. In essence, says Jon Lavallee, enterprise storage architect for FalconStor, the IPStor server acts as a gateway to an assembled pool of storage units on a networkno matter whether it's an IBM, Hitachi or EMC disk array, a hard disk or a virtual tape library. The server takes the entire capacity of the assembled group of storage devices and treats it as one large storage pool. "To the user, it looks like one large disc array, but it might really be a series of different storage devices," Lavallee says.
In doing so, an administrator can assign each application a dedicated storage capacity, such as 50 GB; that storage area could come from one device or be assembled from several devices. A key benefit: The network doesn't need to be shut down to add capacity such as a new server. "It's sort of like running behind a car as it's going down the road and filling it with gas whenever it's required," Lavallee explains.
To back up data under the old AS/400 system, Champion typically had to take its SAP system offline for 14 hours. The new IPStor system employs a different method for backup. In effect, it replicates only changes to the data. Once an initial full-scale backup is performed, the system takes a snapshot of the data changes and replicates those to backup. Now, backups can be performed quicker and more frequently. Snapshots take about one second to perform; thus, Champion's SAP environment no longer has to be taken offline for backups.
A similar situation exists for establishing a so-called Quality Client for testing. Like most organizations, Champion creates a duplicate of the SAP environment to test updates and fixes before bringing it live. On the old AS/400 system, obtaining the Quality Client required at least 14 hours to create the data backup and another one to two days to get it ready for testing. Now, a snapshot is taken and a Quality Client is ready in about four hours.
The data snapshot and replication system is also part of the company's disaster-recovery setup. The IPStor server sends data changes over Champion's MPLS network to a SunGard disaster-recovery facility in Scottsdale, Ariz. After Rita, Champion moved its disaster-recovery site to another city. The hot site in Scottsdale consists of a similar set of Dell application and database servers, as well as 7 TB of Dot Hill storage also orchestrated by an IPStor server.
In an emergency, Etcheverry says company operations can be switched over to Scottsdale and operational in about two hours.
Etcheverry estimates Champion spent about $400,000 for its new disaster-recovery system, including storage, application and database servers, as well as the IPStor component.
The technology team began assembling the components for the project in January 2006, and six months later, on a Friday night in July, turned off the old AS/400 and turned on the new system. Etcheverry says there were a few minor problems in the switchover, primarily related to the transporting of data from the AS/400 to the Wintel system. Despite conducting four data conversion trials prior to the cutover, some errors were missed, Etcheverry notes. He says it's not all that uncommon in a conversion of this size, and over the weekend the bad information was identified, the data reloaded and the problems fixed.
A more serious problem occurred a few weeks later when the new Microsoft SQL database crashed. It turned out there was a SQL bug that could cause the database to fail on high-performance servers. Microsoft had issued a patch, and once the patch was installed, the problem was solved.
With the new SAP platform in place, the Calgary office was brought online in August 2006. Etcheverry's team is now working to bring the rest of Champion's regional offices onto SAP through 2007 and into 2008.
"When we began this project, we started by asking how can we best manage this business going forward," Etcheverry says. "We wanted to keep costs low, but we also needed something that could be scaled cost-efficiently to handle our growth.
"We got that, and in the process we've achieved all of these additional benefits."