Taking Control of IT Changes

By David Strom  |  Posted 2010-06-28

One of the hardest parts about managing IT resources is understanding how to control and document changes to your network infrastructure, applications and systems. And as enterprises deploy more virtualization tools, along with upgrading and improving their desktops, managing these changes effectively can make the difference between satisfying users and dealing with chronic downtime.

Sometimes, it’s a matter of knowing when to stop patching and start swapping an aging system for something that can be better maintained in-house. One example is the city of Ventura, Calif., which had repeatedly changed its 28-year-old legacy system and realized it was time for a replacement. After selecting Agresso Business World to provide its ERP solution, the city had to bring its staff on board with the new technology.

“We had large SAP and Oracle installations and wanted a system that would enable us to transfer expertise as quickly as possible to our own staff to make them self-reliant,” says John Emerson, senior IT executive with the city.

“We live in a dynamic world, and state and federal governments are changing the rules on our operations all the time, and users are changing their needs,” Emerson points out. “We wanted the ability to keep up with these changes without having to hire consultants all the time. We went with Agresso because it had the strongest ability to add files and tables and workflows. We already had a custom system, so this was a critical requirement.”

However, a lot of staff retraining was necessary, and the transition was difficult. “We completely underestimated the difficulty of changing our system because it had been in place for so long that users couldn’t distinguish the system from their jobs,” Emerson explains. The city replaced the outdated character-based system with a “modern application that required using a mouse and Windows. It also took some adjustment to deal with the workflow features.”

Winds of Change

When the Raleigh Durham International Airport built a new terminal, it updated its existing network management and monitoring tools using software from Solarwinds. “What really sold us was being able to quickly build nested network maps,” says Brian de Loureiro, manager of systems and integration for the airport. “Because of the size of our network, we couldn’t represent all our devices on one screen.

“We have different sections of our campus, and with the nested maps, we can click on a building and drill down to the network in that building. We can readily identify if an entire building is down. The mapping makes it quick to identify [specific areas].”

When it came time to upgrade an aging frame-relay circuit, Solarwinds helped the airport’s IT team analyze its Internet connectivity. “We could see the bandwidth we were using and were able to make things run more efficiently,” de Loureiro says.

Previously, the airport used an Excel spreadsheet to keep track of managing more than 4,000 IP addresses across their network, but that had become unworkable. “Issuing IP addresses wasn’t that bad when we had 500 or so,” he explains, “but with the airport’s new terminal, we had multiple contractors that needed these addresses. We had subnets that were being created by subcontractors, and we had a time crunch to open the new terminal. [With the new network tools,] we didn’t have to worry about double-booking IPs.”

Lesson Learned

David Goodman learned the hard way about how and when to involve his IT department in changes made at The International Rescue Committee, a relief organization based in New York City that runs dozens of offices around the globe.

“We had a recruiting package that was bought by another company, and our migration was handled by our HR department,” says CTO Goodman. “The upgrade didn’t go particularly well. The people running the project weren’t IT people, and they didn’t think things through. In the end, it was clear to them that you have to involve the systems people in systems upgrades.”

This doesn’t mean that IT needs to be involved in every project, he explains, but someone needs to ensure that the foundations of requirement-gathering and systems analysis are in place.

“When I was hired, we didn’t have much of an IT department, and many of our staff members didn’t understand the value IT could provide,” he says. “I purposely don’t come into meetings and try to take over a project. I take a more measured approach and have people come to us when they need something.”

Goodman believes getting a business degree and being more sensitive to the business processes helped him sell his services internally. “Our public-facing Website was languishing from a tech standpoint,” he recalls, “so my team and I helped to staff up the Web team and get it to a point where it didn’t need my attention anymore. It’s important to know when to step out of a project and let the end-user department take over.”

Overhauling Documentation

Sometimes, all it takes to make change more manageable is the right kind of documentation, something that Connie Dillard, the longtime IT manager for the city of San Carlos, Calif., knows all too well. In preparation for her retirement from the city this spring, she began an overhaul of their systems and network documentation last year.

“I wanted to make sure that there was an orderly transition when I retired, and it was also time to do a better job at keeping track of our procedures,” Dillard says.

Dillard began using Microsoft’s One Note software to compile electronic “notebooks” full of information, such as user-access lists, project purchase orders, and software- and network-configuration information. “It makes it easier to share this information with the other members of my IT staff, so they can easily and quickly find something when I’m not around. If you don’t document it, you have to relearn how to do some procedure—or call the vendor to walk you through it.”

Dillard uses Screensteps screen-capturing software from Blue Mango Learning Systems to document the process to set up a new server or user account. So everyone on her staff can replay these steps in the future. “This way, it is at our fingertips,” she says. “Since we are still using some software that is more than 10 years old, it’s critical to preserve this knowledge.”

Virtualization has made the city’s documentation requirements a lot more critical, Dillard adds, but not necessarily complex. She created an Excel spreadsheet that keeps track of when individual virtual machines on her host server are restarted, as well as when they need to be patched or updated.

“Not all of us are here all the time, and we needed some way to quickly share what we were doing,” she explains. “This works just fine for our purposes.”

Consolidating the physical servers, along with some other technologies, has made Dillard’s documentation duties a lot easier in some ways. “We used to have paper logs for each of our 31 servers,” she recalls. “That—and having backup tapes and doing the catalogs for them—was very painful. Now we use direct-to-disk backups, and with virtualized servers and automatic logging tools on some of our other software products, we save a lot of time.”

The key thing, Dillard stressed, is to review these activity logs each morning to make sure that any problems are resolved and that what was supposed to happen the night before actually transpired.

Sometimes, the hardest part about changing your applications is keeping your legacy data in a form that you can access once you perform your upgrades. Some upgrades can make this data inaccessible—either because data storage formats become obsolete (remember the eight-inch floppy disk?) or because the newer applications can no longer manipulate the older versions’ data.

That’s what the IT staff members at the Emory University library faced when they received several Apple Macs that author Salmon Rushdie used to write his novels. Software Engineer Peter Hornsby and Senior Engineer Ben Ranker had to develop ways to extract and preserve the machines’ data so that scholars could view Rushdie’s documents.

“That was the first time we had such a valuable collection of computers in our special-collections department,” Hornsby says. “We needed to preserve the context as well as the formatting of the documents.”

As the examples in this article illustrate, the path toward upgrades needs to be managed carefully and with appropriate policies and plans in place. So make sure you analyze at the start the impact your decisions will have on your enterprise and then carefully document all the changes you are making.