Managing Through a Meltdown

By Scott Mayers  |  Posted 2011-12-07

The word “restructuring” can give any IT leader insomnia. But what if you’re the CIO of a major, multinational financial services corporation? What if your enterprise had recently declared Chapter 11 and was being dismantled and reorganized across multiple global locations, functional areas and requirements? And what if that restructuring came in the wake of the biggest banking crisis since the Great Depression?

These aren’t hypothetical questions: This is exactly the scenario that Lehman Brothers Holdings Inc. (LBHI) faced in 2008. Under the leadership of James Johnson, senior vice president of global infrastructure, the company needed to address a long list of complex and interrelated requirements for redesigning its IT infrastructure.

Heading the list was the need to build and manage a data center that met the requirements of a fast-moving, zero-latency trading business without incurring any capital expenditures on the project.


Ready to Turn on a Dime

Most IT organizations don’t face this kind of perfect storm of change every day. But every IT executive knows that rapid, critical business changes can force a quick reaction from the IT organization, often with little warning.

Increasingly, IT has to be ready to turn on a dime, but the infrastructure in place to support business operations one moment might be considered ill-suited for the organization that’s in place after the change. Although no one can foolproof IT design for the future, you can prepare yourself and your organization for unforeseen changes by adopting the following best practices.


1. Think outside the cloud.

When it comes to designing infrastructure solutions that address big business-change events, many vendors seem to think that just saying the word “cloud” is enough. Although cloud-based technologies offer tremendous promise in terms of scalability, flexibility, speed-to-deployment and cost-effectiveness, those clouds have to be tethered to the earth in order to have a measurable impact on the business.

In the case of LBHI, the company needed to “rent” technology infrastructure rather than buy its own equipment because of a very simple, but very restrictive, constraint: Since it was in Chapter 11, the firm couldn’t expend capital on infrastructure.

Therefore, the team had to get creative in order to design a data center environment that met a long list of requirements without counting as capex. Specifically, the company had to do the following:

• Install separate and flexible data center environments.

• Configure high-performance servers and storage.

• Design and install network IP telephony, as well as audiovisual and wireless systems.

• Design and deploy workstation builds.

• Migrate and implement business applications.

• Create and support policies, procedures, applications and end users through an ITIL (Information Technology Infrastructure Library) IT service desk.

• Establish a proactive 24/365 network operations center.

• Conduct inventory and track IT assets.

2. Take a “cascading priorities” approach.

Traditional architecture strategy, even in the design-build environment, tends to put threads of activity into silos. This means that it might not be until the end stages of a major infrastructure project that gaps and conflicts between silos become apparent. This approach should be avoided.

In the LBHI project, each item on the requirements list was interrelated, which meant that the company had to cascade the priorities, mapping out how each one would affect the next and designing the overall architecture to support a “waterfall” of processes and activities across the servers and applications.

At the same time, the company was adapting to a changing financial environment in which business still had to continue as usual. This meant that not only did the technology need to be effectively put in place, but maintaining support mechanisms was imperative, as well. In other words, the design-build process had to be disaster-proof.

As the transition got under way, an on-site, private VMware cloud computing platform delivered speed and flexibility for time-sensitive applications and data, while less critical applications and data were migrated to an off-site location. This hybrid approach maximized the benefits of both solutions: low-latency computing through on-site technology, combined with cost savings from off-site storage. To ensure business resiliency, all locations were also replicated to a disaster recovery site.


3. Measure twice, test infinitely.

Infinite testing might sound expensive—and impossible. But using a testing-oriented approach to IT infrastructure redesign during business-change events is critical.

First of all, the goalposts often move—quickly and far—during a major change. So you’ve got to be sure that the results you are measuring actually hold up when testing against the objectives you are trying to achieve.

Second of all, testing isn’t merely an opportunity to confirm that all is working; it’s also a chance to see what’s not working and how to make changes for the better. In the case of a financial services company, the tolerance for redundancy, recovery and incident-response processes is zero, so constant testing would reveal system failures that put the business at risk of regulatory exposure.

Designing smart systems is just one side of the coin. Testing is the other, and it requires at least as much time and attention.


4. Adopt an adaptive mindset.

Thankfully, most companies don’t have to face the challenges associated with Chapter 11. Companies do, however, face major change events every day: mergers, acquisitions, divestitures, new-technology adoption, new senior leadership ... the list of possible changes is just about endless. Even the smartest companies cannot anticipate every change, but those that adapt best to change will be more successful than those that are entrenched in one—often old—way of doing things.

Just as an organism’s ability to adapt to change is essential to its survival, an organization’s ability to adapt to change is essential for it to maintain its competitive advantage. The cliché “the only constant is change” might sound trite, but it’s an important truth when it comes to IT infrastructure.

With an adaptive mindset, the IT organization can move at the speed of technology while evolving at the speed of business. That is truly forward thinking.


Scott Mayers is a principal and the director for the Integrate and Manage practices at Align, a consulting firm that enables business and IT alignment and delivers solutions for business change and growth.