Cool Cash

By Brian P. Watson  |  Posted 2007-10-29

Going "green" has become the initiative du jour for many companies, as environmental campaigns trumpet eco-friendliness and businesses face heightened energy costs. For Wells Fargo, the San Francisco-based financial services firm with $540 billion in assets, the effort began four years ago, before the hype, when growth forced the company to expand its data center capacity.

But it wasn't all about the environment: Bob Culver, senior vice president of Wells Fargo's technology information group, says the firm's top priority was to ensure security and availability of the services it provides, which include consumer banking, investments and mortgages.

In the process of building two new data centers, though, Wells Fargo put in place a number of energy-saving features to lower power consumption and reduce costs. It accomplished both—and there hasn't been an outage yet.

"They've performed the role we expected them to perform, so there haven't been any surprises," Culver says.

The firm opened new, high-density facilities in Minneapolis (in August 2005) and Tempe, Ariz. (about six months later). Both facilities house 7,000 to 8,000 servers from Dell, Egenera, Hewlett-Packard, IBM and other vendors.

Standardizing on a particular brand has not yet been an option, Culver says. Over the years, Wells Fargo has developed purchasing agreements with various vendors, particularly Hewlett-Packard and Dell, and also has an enterprise agreement with IBM that includes mainframes and servers.

But consolidating with one vendor won't be considered while all those varied servers are still operational, he says. On top of that, certain business lines required specialty systems, leading to more vendors entering the fray.

The Big Chill

The planning process began in 2003, when Culver and his team worked with contractors, architects and consultants to design the facilities. Wells Fargo hadn't had excessively high energy costs in its other data centers, but with the new construction, Culver saw an opportunity to explore measures to lower consumption. That's become standard operating procedure for builders, so he and his team decided to run with it.

In the Minneapolis facility, Culver had to decide which kind of economizer to use. (Economizers are mechanical devices that help regulate energy usage.) One option was an air-side economizer, which uses air from outside the facility to cool the environment inside. But there was a downside: That would mess with the humidification needed to keep the air from becoming too dry, Culver says.

Instead, Culver turned to water-side economizers, which use evaporating water from a water tower to counteract heat emitted from servers. This method, known as "free cooling," let the company forgo chillers—machines that cool the water used to dehumidify the facility. That saved the company 300 kilowatts an hour per chiller, or about $150,000 a year so far, which Culver expects to double in the next two years.

The firm also looked into floor-level cooling for the Minneapolis data center. Many data centers use self-sustaining air conditioning units scattered around the floor and linked to chillers to distribute cool water. But Wells Fargo had eliminated the chillers, and for the density of servers in the facility, Culver and his team decided to go a different direction.

The engineering firm Wells Fargo had contracted for this project had built a central fan system for American Express, which had a lower-density data center. So Wells Fargo had the firm adapt that system to its new facility. The system is computer controlled and lets facility managers monitor how powerfully the fans operate at any given time. "It allows us to put the air where we need it," Culver says. "The system only works as hard as it needs to because it's regulated by the computer system."

And, he adds, it saved the company about 15 percent in power consumption.

In Tempe, though, Culver and his colleagues had to cope with an entirely different climate. Without Minneapolis' constant cooler temperatures, the Tempe facility wasn't a candidate for free cooling. So here the company used the modified floor-cooling system with variable- speed fans and chillers.

The controls in place at the two facilities have yielded impressive performance results: Both data centers have had 100 percent uptime since they went into operation, Culver says.

Powering Ahead

Now the firm is aiming to construct a LEED-certified data center about 12 miles from Tempe. (LEED, leadership in energy and environmental design, was devised by the nonprofit U.S. Green Building Council as a set of benchmarks to measure a structure's environmental friendliness. The council rates buildings based on sustainability, water efficiency and other factors.)

But it won't be easy, Culver concedes. "That's very difficult to achieve with data centers because they do consume so much energy," he says.

Culver plans to rely on dynamic power management features from Hewlett-Packard. The software, which comes with HP's ProLiant servers, lets administrators shut off power in semiconductor chips until it's needed. Culver says Wells Fargo will begin piloting the tool in the coming months.

The company will also have to contend with higher material prices than it encountered while building its data centers in Minneapolis and Tempe. Culver wouldn't reveal the exact price tag for the facilities, but he said Minneapolis cost less than $100 million, while Tempe came in "well under" that.

Then, copper—an essential component of data center wire and cable—was selling for about $.70 per pound. Now, Culver is looking at prices around $3.50 per pound. On top of that, the Environmental Protection Agency has sunsetted older power generators as it has established new emissions standards.

Manufacturers are ramping up production, but purchase costs have increased.

To build the Minneapolis or Tempe facilities today, Culver says, costs would exceed $100 million. He's aiming to keep expenses for the new Tempe data center less than or around $100 million.

To accommodate heavy transaction volumes, financial institutions are constantly updating and renovating their data centers. In planning, power consumption and costs have become a higher priority for these firms. But for Culver—as well as many of Wells Fargo's competitors—other concerns come first.

Protecting sensitive customer data is the top priority, Culver says. Next comes availability.

"You tackle the purpose at hand, and design something that has security and can be available," he says. "Then you talk about making it most efficient: How can I save energy on mechanical and electrical systems?"

Wells Fargo also plans to expand its virtualization projects. But that will require overcoming cultural obstacles, according to Culver. More than two years ago, the company invested in VMware's ESX Server, a hypervisor-based virtualization software tool that provides automatic failover and live backups. As of second quarter 2007, Wells Fargo had virtualized almost 500 servers.

Still, the virtualization initiatives were slow to take hold. The biggest problems, Culver says: Some lines of business didn't want to share equipment with others. Overcoming that is no easy task, because Wells Fargo has hundreds of business lines organized into 10 or more groups.

Culver says the company also plans to explore more green techniques, though he's concerned about payback. An experiment with a solar array on the roof of the Tempe facility is under way, for instance; Culver is not entirely sure about the potential return, he says, but this is a low-cost investment worth trying. Other eco-friendly features such as motiondetector lighting and variable-speed fans also cost little but can yield significant returns.

Before considering any other new tools or techniques, according to Culver, Wells Fargo will examine the potential return on investment. "If someone says the incremental cost of a project is $5 million and the return is 23 years, we're not going to do it," he says. "It's still a financially driven motivator for us."

Wells Fargo At a Glance

420 Montgomery St., San Francisco, CA 94163

(866) 878-5865

Offers a range of financial services, including consumer and corporate banking, insurance, investments and mortgages

Senior Vice President, Technology Information Group:
Bob Culver

Financials in 2006:
$48 billion in revenue; $8.5 billion in profit

Plan and build two new high-density data centers without increasing energy expenditures on the company's other facilities.