Turning A Mountain Into a Molehill

By Thomas Boyce  |  Posted 2008-02-21

By Thomas Boyce

One of the primary goals of the federal government’s National Institutes of Health (NIH) is to foster fundamental scientific discoveries and innovative research strategies that advance the United States’ ability to protect and improve human health. The NIH advances this goal by providing more than 55,000 grants to research scientists around the world—grants that consume more than 80 percent of the institutes’ annual $28 billion budget.

Since the 1960s, the NIH has provided automated support for processing and administering these research grants—not only for the NIH, but also for a broad spectrum of federal agencies including the Centers for Disease Control and Prevention, the Food and Drug Administration and the Veterans Health Administration. Since its inception, the system, known as Electronic Research Administration (eRA), has been transformed several times to take advantage of four decades of technology advances.

When I joined the NIH as eRA program manager in May 2005, the system was at a major crossroads. The eRA system—24 complex applications linked through a shared 6 TB transactional database—appeared to be functioning successfully, supporting the processing of tens of thousands of paper-based grant applications each year. In reality, the eRA system was on the verge of a very public crisis.

Between 1998 and 2003, the NIH budget had doubled, resulting in a massive workload increase for the eRA system, with grant applications skyrocketing from 40,000 to 80,000 per year. Despite these increased demands on the system, eRA staff members were charged at the same time with converting the system to an entirely Web-based “eSubmission” environment that would eliminate paper processing altogether.

It was clear that we faced significant challenges in achieving program goals. Fortunately, I had inherited a dedicated staff that was used to making heroic sacrifices to avert failure. My goal was to improve processes so heroics would become unnecessary. Here’s how I did it.

First, Re-evaluate Existing Processes

In 2004, the NIH noticed that substantial increases in the eRA program budget were not yielding expected results. In fact, the frequency of both missed software delivery deadlines and system outages was increasing at a disturbing rate.

Despite these concerns, senior management announced in August 2005 that NIH would lead the government’s migration to an all-electronic grant-application process. This new system, one piece of the larger eRA program, was dubbed eSubmission. It required converting our systems from a paper-based process, which relied on data entry and scanned applications, to a system that would accept the equivalent of billions of pieces of paper annually via an XML data stream.

My staff assured me they had been planning for years for the advent of electronic applications. In fact, the system had already accepted almost 200 applications electronically via a pilot program. Nevertheless, with the first production round of 1,800 or more applications only three months away, bugs were still common and substantial portions of software required rewriting.

The need for a fundamental process change quickly became evident. At one of our weekly status meetings, I asked about interdependencies and the master project schedule. After the meeting, a lead staff member took me aside and whispered, “We don’t have an integrated project schedule. In fact, we don’t really have individual project schedules. We create these status reports from staff input on a weekly basis.” When I asked how this was possible, given the size and complexity of the program, the staff member said he had asked the same question when he joined the program, and that I would “get over it,” as he had.

That’s when I began to examine program procedures and delve into the processes being followed. I discovered that the program did have a procedures guide, but there were two major problems. One, it was overly complex. Two, it existed mainly as shelfware: Few people were aware it existed, and no one used it regularly.

I immediately implemented a simple yet effective risk management approach using Excel spreadsheets. We relied on rankings of probability (high/medium/low) and likely impacts. It was essential to engage the staff in identifying areas that needed immediate attention and to introduce more structured approaches to running the program.

In addition, I announced that, within several months, all staff members were to begin using Microsoft’s Enterprise Project Server to record the time they spent on various projects and to verify contractor invoices against tasks and time recorded. I was struggling to identify all the program activities, and felt we could take a huge step in the right direction by getting all 300-plus employees to record their activities.

Contractors helping us implement this new policy said my execution timetable was unreasonable. They argued that they needed time to identify the most likely activities across the entire program and develop project templates before we could move forward. Nevertheless, we managed to produce the first iteration of Project Server on time.

Furthermore, I placed limits on the scope and schedule of our software releases. Until this point, my team and the NIH had boasted about their ability to release new software daily, claiming that this release rate reflected their flexibility and agility. I was convinced—and was later proven correct—that this daily rate was symptomatic of the lack of process control.

We used the eSubmission project as a pilot for the implementation of these new procedures. Although the scope of the software revisions was extensive, the project team quickly began to see fewer bugs and was forced to deal with far fewer changes. The success of the eSubmission project led other departments to begin adopting some of the new practices.

Near Disaster: Budget Overload

Though it was moving along well, the eSubmission project was consuming large chunks of staff time and quickly outpaced the budget. This prompted closer scrutiny of the entire budget and led to the discovery of a major problem.

Over the previous three years, the program budget had grown steadily and management, anticipating continuing increases, had added staff. Now, however, budget increases had ceased, and management was predicting flat budgets for the next several years. As a result, program expenditures were running thousands of dollars more per day than could be supported by the approved budget.

I had been meeting regularly with the program’s steering committee, providing updates on issues and problems. I discussed the budget problem with the committee and asked that the eRA program be allowed to focus exclusively on two projects in the coming fiscal year—instead of the six major initiatives being worked on—in addition to the eSubmission project. I also made it clear that, even if this proposal was accepted (and it was), I would still need to make substantial reductions in staff to work within the approved budget.

There were two paths to achieving the staff reductions. The first—which could be implemented fairly quickly and easily—was to eliminate some of our 20 existing contracts. The drawback was that we would lose valuable talent.

The second option was to surgically reduce staff across the program, ultimately cutting 20 percent of existing headcount. While this route would take longer, it would better enable us to retain our most prized contributors.

After obtaining approval for the surgical approach, I engaged eRA staff to determine how best to address this challenge. I sought input from at least three staff members of every contractor working for the program, asking for candid evaluations of contributors—who was a marginal performer and who was the one person they couldn’t afford to lose.

In addition, I met with each contractor, explained the budget crisis and asked them to restructure their costs. Most of the contractors complied, despite knowing we would still need to reduce overall contract staff by 20 percent. Ultimately, we were able to save nearly $2 million a year through cost restructuring alone.

One of the final challenges we faced was to revamp our entire approach to software development. We had seen benefits from taking a project-based approach with eSubmission, and the goal was to effect these changes across linked projects and the entire system.

The key to building processes the staff would endorse was to empower them to define what was important. The first procedure they drafted was how to raise an issue with any process and get it changed. The staff then followed a structured software lifecycle without over-engineering the process. The result was early acceptance and adoption of new development processes across the entire program.

No Pain, No Gain

The past two and a half years have been tough, but rewarding. Given the chance to do it again, I would definitely start with the budget, making sure I understood the program from that perspective first. I would also freeze all but the most critical projects, taking a broader, portfolio view of our investments to determine how our priorities aligned with the interests of the program’s steering committee and stakeholders. In addition, I wouldn’t be so quick to implement Microsoft Project Server, because my approach prompted us to change course several times as we pinpointed root causes and more promising solutions.

In retrospect, I should have allowed more time to make sure staff members understood proper planning, the essential review points in the life of a project, and how to build and track projects within a more structured environment. The main lesson I learned was to develop and understand the processes first; then choose a tool to help implement and enforce those processes.

Finally, engaging and empowering the staff to identify challenges and develop solutions was absolutely critical to our success. The previous management team was risk-adverse: They simply did not want to hear about problems, preferring instead to “shoot the messenger.”

This attitude had contributed significantly to the dysfunction I discovered when I started the job. Once the staff determined that I was open to hearing about problems—potential or existing—I had a line at my door. Our discussions helped me realize not only the full scope of the challenges we faced, but that I had a dedicated staff willing and able to help. All they needed was an opportunity.

I can honestly say I have never worked as hard in my entire career as I have during the past 30 months. In preparing to move to another government agency and new challenges, I find myself asking whether all the hard work was worth it.

Two separate meetings in recent weeks answered that question. In one, I met with the NIH senior management group that oversees the eRA program to prepare for the 2009 budget request. In reviewing the current eRA budget and priorities, several in attendance asked why the program was being nickeled and dimed in light of all the success it had achieved during the past two years. Clearly, it was a sign that what my team and I did was producing tangible benefits.

More rewarding to me personally was a recent preliminary design review meeting—just one of the new process gates the staff must go through in preparation for our enterprise software releases. There was a heated exchange about how to improve the project templates and the design review report to make them easier to generate and more meaningful.

It was clear that the staff had gotten past their resistance to a new way of doing things, and had embraced my continuous-improvement approach. I took that as a sign of progress toward achieving my ultimate goal—smart change management.