Corporate Express: Then, There Were Two

In 1998, things were looking bleak for Corporate Express. The world’s largest business-to-business office products company had gotten that way through hundreds of acquisitions. It had taken on too much debt. In November of that year, Standard & Poor’s downgraded the company’s credit rating to junk.

PDF Download

Monty Sooter, Corporate Express’ chief information officer, had been charged with trying to integrate the operations of the 300-plus acquisitions his company had made. Now he was inheriting a new mission—slashing the cost of the company’s data center operations.

Sooter and Andy Miller, the company’s vice president of technical architecture, decided on a path being taken by a growing number of companies: server consolidation.

By replacing dozens of smaller servers with a small number of large ones, companies as diverse as First Union National Bank and cosmetics seller Mary Kay hope to reduce hardware, maintenance and administrative costs, and recentralize their data center operations.

On the whole, server consolidation seems to work in achieving these ends. Mike Chuba, an analyst at Gartner Inc., says consolidating servers properly can save 20% or more on costs. “But I’ve seen projects that were done wrong, and ended up costing the company more than the servers they already had.”

Corporate Express managed to do better than 20%. After it replaced 43 Hewlett-Packard servers with two powerful Sun servers in 1999, the cost of hardware in the company’s data center dropped from $375,000 a month to just $75,000 a month—an annual savings of $3.6 million, or 80%.

The inefficiency that allowed Corporate Express to make such an outsized improvement came about for the usual reasons. As vice president Miller Says, “Every time we had a new development project, we’d buy a new server.” Each application—from the company’s PeopleSoft human resources software to its custom-built e-commerce and order management applications—had at least one dedicated HP 9000 multiprocessor server.

Because of the ad hoc way in which the servers were acquired, almost all were underutilized. This is not atypical in a distributed computing environment.

Miller had been working to open up the company’s infrastructure options since his arrival in May 1997. In particular, his goal was to make sure the company’s applications ran on standard platforms, like Java, rather than only on HP’s version of Unix.

Miller’s team got a push from Sun, which was eager to replace HP as the vendor of choice in Corporate Express’ Broomfield, Colo., data center. Several months before the server consolidation decision was made, Sun came in to show how Corporate Express’ custom applications could be moved to the Solaris operating system. Miller says it only took about two weeks to get working versions of the applications compiled and running, though there were a few rough spots along the way—including bugs that made the system crash.

On the whole, though, the migration went smoothly—mostly because Corporate Express’ custom code had been written for standard Unix platforms. It was only the packaged software—software from Oracle, say—that was optimized for HP-UX. “Oracle obviously moves easily from HP-UX to Solaris,” says Miller.

After talking to both Sun and HP about their options, Sooter and Miller went with the Sun proposal: replacing the 43 HP 9000’s with two Sun Ultra Enterprise 10K servers, each configured with 10 UltraSparc processors. The reason for the choice was straightforward, according to Miller. A Sun UE10k server could be divided into a number of virtual smaller servers, divvying up portions of the server’s memory and processors for specific tasks. Hewlett Packard wouldn’t have such a “partitionable” server until 2000, when it introduced its Superdome systems.

“What was happening at that time was Sun had a partitioning solution, while we were selling clusters,” says Vish Mulchand, server product line manager at HP. Even so, Mulchand contends that the UE10k wasn’t necessarily a better technical solution. “They could have consolidated to fewer, newer [HP 9000] systems. But 1999 was the peak of the dot-com era, and while I don’t want this to sound like sour grapes, Sun was the market leader. They may have done it for valid reasons, but they must have also gotten caught up in some of the Sun hype.”

The consolidation project kicked off in april 1999. It pulled in 12 members of the Corporate Express technology team full time, with an equal number of consultants and technicians from Sun Professional Services onsite for the duration of the move (Download PDF above to see chart, “Saving 80% on Hardware Costs Through Consolidation”).

While the process of moving applications turned out to be relatively simple, the real work involved moving the data that went with them—approximately 4 terabytes of it, including customer order records, payroll, and other critical operating information. “It took us two months to figure out how to move everything,” says Miller.

The plan was to use as much as possible of existing storage infrastructure—a combination of EMC and Hewlett-Packard disk storage, attached directly to the old servers. The disks would be added to a storage area network connected to the two new servers by a storage switch from storage network vendor Inrange.

One problem: The file systems of HP-UX and Solaris weren’t compatible, meaning the disks could not just be switched from one server to another. “We had to move things physically from one environment to the other,” says Miller. Everything had to either be backed up to tape and restored on the new servers, or copied across the data center’s network.

The biggest headache was the 2.5 terabyte data warehouse. “It was difficult to move just because of its size,” Miller says, “so we had to do parallel loads, running it on both HP and Sun as we moved the data across. It took three weeks to move the data, because we couldn’t have downtime.”

The backup system was an issue, as well. The old system had consisted of HP tape libraries connected directly to servers, and Unylogix’s Alexandria tape library software; backups were done over the center’s 100 million-bit-a-second network.

“We had to buy new tape backup libraries, and bolt on a new backup software system,” says Sooter. With the shift to Sun, Corporate Express chose Veritas backup software, but it didn’t install a storage area network-based library. Instead, it installed a StorageTek library using a bridge based on Fibre Channel, an extremely high-speed networking technology. The library was configured with only 20 tape drives. “We wanted to optimize use of libraries so they would last us for the duration of the lease,” says Miller. That meant a little more time dealing with unfamiliar territory for the configuration team. The upside: Sooter and Miller managed to cut the number of tape drives by more than half, since libraries were no longer dedicated to specific servers.

Fewer tape drives caused Corporate Express’ data center personnel more than a little culture shock. “It makes the logistics of figuring out where to hook everything up more difficult—you have to worry about backup windows and things like that,” says Miller. “It was a big transition for people.”

The consolidation of servers also meant that, for the first time, applications would have to share a server. Initially, the new servers had only 10 central processing units (CPUs) each, reducing the total number of processors in the data center by nearly 75%. “Every application team was used to having its own server,” says Miller. “There was lots of anxiety about what would happen with apps running on one hardware base. But the only way to get real value out of [a server consolidation] is to drive utilization up. Our philosophy was to keep utilization as high as reasonable.”

Once things fell into place, Sooter and Miller found some pleasant surprises. “The good surprise was the performance,” says Miller. Depending on the application, response time was twice that of the old server configuration. The PeopleSoft batch process performance, for instance, improved by 98%.

In the end, the project took four months to complete, wrapping up in July of 1999. And that, it turned out, was just in time—that same month, Corporate Express announced it was being acquired by Amsterdam-based Buhrmann NV, a huge player in office supplies.

Buhrmann decided to make Corporate Express its office products brand, and merge all of its operations in North America into the company. As part of that process, 12 data centers Buhrmann had previously operated in the U.S. were shut down and their operations moved to Colorado. To handle the additional load, the servers in Broomfield were expanded to 56 CPUs and 56 gigabytes of RAM, and eventually two additional servers were brought in.

Alas, things didn’t go completely as planned with the servers. Miller says the company experienced reliability problems with Sun’s UE10k because of a now well-documented problem with the UltraSparc CPUs they came with—they lacked error-checking and -correcting memory in their cache. The result: partitions within the server would crash unexpectedly. “We ended up having all of our CPUs replaced,” says Miller.

Last year, Buhrmann acquired U.S. Office Products, a Washington, D.C., company, further expanding the consolidation requirements of the Broomfield data center. Over the past three years, over 600 local facilities have been closed down. While the Broomfield data center only handled 15% of the company’s operations in 1998, “now we have 90% of the company’s business running through this infrastructure,” says Sooter.

Corporate Express is now in the process of another consolidation: migrating to a Sun Fire 15K server with 72 processors and “around 192 gigs of RAM,” says Miller. “We’re taking 144 of our CPUs down to 72,” consolidating the capacity of nearly three of the UE10k’s into one new server—and bringing the total server count back down to three.

“We could not have scaled up as much with the old architecture,” says Sooter. “We would have been forced to do something drastic just to continue our business. The consolidation ended up being a big benefit.”