New Orleans: Picking Up the IT Pieces
The day after Hurricane Katrina raked through New Orleans with 125 mph winds, a raiding party led by Greg Meffert, the city's technology chief, ventured out of the battered Hyatt hotel across the street from City Hall.
The streets were slowly but steadily filling with water from burst levees, and before long 80 percent of the city would be underwater.
Since the time the storm had hit, early on the morning of Monday, Aug. 29, Mayor Ray Nagin and his staff had been subsisting with practically no communications beyond an overloaded police radio network.
Cell phone towers had been knocked out, and most landline phones were dead, including those in the Hyatt. The satellite phones they'd brought into the hotel proved useless, after the phones' batteries malfunctioned.
As he headed out in a National Guard Humvee with his colleagues, Meffert had a plan to get Nagin back on the phone with the outside world so the mayor could call for help.
Earlier, the city's Web project manager, Scott Domke, had managed to establish an Internet connection off what was left of the Hyatt's network. That was a start, but Meffert needed more.
While the likelihood of a hurricane the size of Katrina hitting the New Orleans area had long been predicted, the threat was never deemed immediate.
A good disaster recovery plan, however, is well structured and rehearsed as if the event had taken placeMeffert's team, for example, could have done better by testing ahead of time the satellite phones they had been assigned, instead of finding out they didn't work when it was too late.
And while New Orleans did have an overall emergency response plan, which, for example, specified that the city would maintain lists of the phone numbers of critical personnel, the emergency plans placed too much faith in phone service remaining intact, Meffert says.
Residents trying to call 911 had their calls dropped after telecommunications switches dedicated to the emergency service failed.
Radio communications between police, fire and other emergency-response agencies were hampered by incompatible frequencies and equipment between state and local agencies, for example.
And although there was an emergency communications system that was supposed to bridge the radio frequencies used by state and local first-responders, it in turn was dependent on a landline telecommunications circuit that failed.
Meffert says his experience shows the weakness of relying on what he calls "hard-wired, 1950s technology" like traditional phone systems rather than more flexible Internet-based communications.
"The model still assumes that there will be one central number that everyone knows to call," he says.
Outside of a time of crisis, it's also easy to make unrealistic organizational assumptions, he explains, thinking that "even at this time of crisis, everyone's going to be sitting around the same table, stroking their beards, trying to decide what to do. That's not just wrong, kind of wrongthat's way wrong."
In reality, key players were scattered and often unable to talk with each other.
According to Meffert, the biggest mistake officials at every level made was treating disaster preparedness as a long-term problem that could be left for another day.
"Before this, the worst-case scenario always had sort of an ethereal quality to it," he says. "It was a virtual problem. And usually that gives you a virtual solution, rather than a real one."
In the six months since the hurricane struck, Baseline has been studying the role of technology in the crisis and recovery, spending time in New Orleans in November and January, interviewing city officials, and talking with technology and disaster recovery experts who have been assisting the city.
Much of what's admirable about the technology team's response was improvised, rather than planned. That's not surprising given the scale of the disaster, says Unisys disaster recovery specialist Ed Minyard, who assisted the city's recovery efforts. "Mike Tyson said everyone has a plan, and then you get punched," he says. "In the middle of something like this, 80 percent of what you think you know is questionable, and the other 20 percent is just dead wrong."
On that Tuesday morning, Meffert's destination was an Office Depot at the corner of St. Charles Avenue and Martin Luther King Boulevard.
That happened to be a location where one of the wireless surveillance cameras Meffert's technology team had previously deployed as part of a crime-fighting initiative hung from a light pole. But when he arrived at the Office Depot in the Humvee, accompanied by Domke and Police Chief Eddie Compass, Meffert was ready for some larceny of his own.
"It had already been looted by the time we got there," Meffert remembers.
Goodies like digital music players were long gone, he says, "but they didn't take the geeky router stuff."
And that was what he was aftera way of turning a single Internet connection into multiple VOIP (voice over Internet protocol) phone lines.
Another of the raiders, Domke, had brought his own shopping list to Office Depot.
The night before, Domke had preserved the Web site as a channel for crisis communications.
In the early hours of the storm, Domke was able to post updates to the Web site with a Verizon Broadband networking card, using a cellular Internet connection to reach the city's relocated Web server in Dallas.
But before dawn Monday morning, the storm knocked down too many cell phone towers and the connection went dead.
But Domke had another resource at his disposal. He had bought a Vonage phone about a week before, intending to use it at home.
Vonage provides a VOIP service aimed at consumers who want to make cheaper long distance calls over a high-speed phone or cable Internet connection.
The phone itself was sitting in the back of his car on a waterlogged city street, but he had gotten as far as establishing an account.
Quizzing the hotel staff, Domke learned that there actually was still some telecommunications service to the building, even though the hotel's internal phone system was offline.
Purely by chance, the mayor's staff had camped out in a conference room just down the hall from a telecommunications wiring closet, which was getting power from a generator.
So, Domke was able to get an Internet connection by plugging into a network switch in that closet and snaking an Ethernet cable down the hall.
Domke didn't have the Vonage SoftPhone software required to make calls from his laptop, and what he really wanted anyway was a way to make and receive calls over regular phone handsets.
Even later, however, after the Hyatt restored partial phone service, the hotel system wasn't really appropriate for this emergency because calls had to be manually connected through a switchboard.
When the city was trying to connect with the White House, for example, it needed to be able to give out a direct dial number where officials could return the call.
A quick tour of the Vonage Web site showed that what he really needed was one of the Vonage routers made by Linksys, which has an Internet connection on one end and standard analog phone plugs on the other.
The order-online option was no goodFedEx wasn't exactly making regular deliveries to New Orleansbut the Web site also let him look up the nearest reseller, which led to the raid on Office Depot.
"The way the water was rising, we thought we might not get another chance," Domke says.
He, Meffert and Compass reached the Office Depot at sunset and entered the store in the dark. Compass provided cover, chasing away looters who tried to re-enter the store.
Domke went shopping for the Vonage equipment he needed, while Meffert rounded up other supplies. They also discovered a Cisco router in the back room that they thought they could use.
For lack of a screwdriver, they tried to use butter knives to remove it from the server rack. When that failed, Compass simply ripped it out of the wall.
The router was later used to restore e-mail service at City Hall, providing another way for officials to rebuild communications.
When they returned with the loot, Domke built a small network from the routers that allowed the mayor's staff to make their first outbound call just after midnight.
Following another raid on Radio Shack to round up more analog phones (those in the hotel were digital), he and other members of Meffert's IT team eventually got eight phones working off the single Internet connection.
As a result, city leaders stepped up their calls for help, and Wednesday night President Bush called back while returning from his Texas ranch.
That's when Nagin got on the line and "told him we had an incredible crisis here and that his flying over in Air Force One does not do it justice."
For Meffert, Domke and others who'd begged, borrowed or stolen basic supplies in order to connect with the outside worldand seen the crisis firsthandhelp couldn't come soon enough.
IT plansbefore Katrina">
Boating through flooded streets and personally pulling people from the water was not exactly what Meffert thought he'd signed up for when he joined the administration of Mayor Nagin following a 2002 city election.
A former software executive, Meffert became the city's first technology chief, and in 2003 he was given an expanded role overseeing the city's engineering and land management departments.
Today, his titles include chief technology officer, chief information officer and deputy mayor.
Nagin had campaigned partly on the idea of improving city government through the strategic use of technology, and he enlisted Meffert to serve on his transition team after his election.
"People tend to assume, because of the role I have right now, that I was close to the mayor," Meffert says. "Actually, I did not know the mayor before I took this job." But Nagin knew Meffert by reputation.
In the city's tiny technology community, Meffert was famed for having raised some $35 million in outside investments in his startup, a document management firm that evolved into a specialist in encrypted document transmission over the Internet. By 2000, however, he had stepped aside in favor of another CEO.
Known at various times as ITS, NetEx and Certia, the company moved its headquarters to Virginia and got acquired.
Meffert initially stayed in New Orleans as its CTO, but by the time of Nagin's election, he was looking to get out. He signed on for $150,000 a year, well below what he'd earned in the private sector.
In his new job, Meffert aggressively pushed for the adoption of new technology. Initially, his mission was to save the city from another kind of disasterits own inefficiency and reputation for corruption.
When the police raided a city permits office suspected of peddling favors, he and his team were there to help the cops recover clues from the office computer systems.
That was part of a crackdown that eventually resulted in 84 arrests of city workers.
He and Nagin approached the city as "a classic BPR problem," he says, referring to the business process reengineering strategy of redesigning work processes for greater efficiency and using technology as a way to force change.
At the time, the city's existing management information systems department was "all about running mainframesthat's all they did. That, and very old phone systems," Meffert says.
He began migrating city systems toward client-server or Web architectures, primarily built around Microsoft technologies.
The city expanded an existing $25 million outsourcing agreement with ACS, a Dallas-based computer services firm that works with state and local agencies.
The city's mainframes, which Meffert wanted to phase out, were moved out of the data center on the third floor of City Hall and into an ACS data center in California.
To better manage city finances, Meffert bought and customized Microsoft's Great Plains software package (recently re-branded Dynamics GP).
Great Plains is primarily known for serving small to midsize businesses, but Meffert put developers to work tailoring it to city functions.
Knowing that the kind of people he wanted to employ were unlikely to be attracted to civil service pay grades, Meffert contracted out for technical talent through Ciber, the Colorado-based systems integrator, and a local subcontractor, Imagine Software.
Imagine effectively became his project management office, supervising any other contractors or service providers.
By making this an ongoing relationship, he wanted to provide the continuity for a systematic approach to the city's technical infrastructure.
The city's pattern in the past had been to hand out one-off contracts with little oversight and let them drag on for years, he says.
His goal was to provide the city with much more modern and flexible systems, and to generate positive cash flow in the process.
After only a few months on the job, he showed how more rigorous data analysis could boost the city's revenue by millions of dollars.
By matching occupational licenses against sales tax data, he revealed how many businesses were routinely skipping sales tax payments. No super science involved, just a database match, but it was something that hadn't been done before.
When he got around to replacing the city phone system with Cisco VOIP gear in 2004, Meffert pointed out that the city had been paying more than $1 million per year to rent phone equipment, plus about $800,000 per year for service calls for routine tasks like moving extensions.
In contrast, the Cisco equipment would be purchased through a five-year, $1.2 million per year rent-to-own contract, and the easily reprogrammable phones would eliminate most service calls.
According to estimates prepared before Katrina, the city was on track to improve its finances by up to $50 million by 2006 as a result of technological improvements.
The city was expecting to bring in an additional $7.5 million in sales taxes, plus $1.7 million in interest on unpaid taxes, while saving money in areas like telecommunications contracts ($9.5 million) and computer operations ($2 million per year).
When Meffert's developers overhauled the city's Web site, they also built their own custom Web portal and content management system.
More important, they began to turn the Web site from an online brochure for the city to an active communications tool and a way for residents and businesses to get services from city government.
Before this effort, New Orleans was at the bottom of the rankings in the Best of the Web contest run by the Center for Digital Government.
But in judging that was completed just before Katrina, the center gave New Orleans first prize (a tie with Washington, D.C.) based on the number and quality of government services offered online.
By that point, residents could pay taxes, report potholes and other issues, register a business, apply for a business permit or look up property assessments online.
"Going into this storm, I thought that was my capper," Meffert says. "I was, frankly, ready to get up on the horse and go back into the private sector."
The danger that a major hurricane could pose to New Orleans was well known.
In 2002, The Times-Picayune published an exposé called "Washing Away" that predicted most of what went wrong in 2005, including the widespread failure of communications systems, and questioned the integrity of the levees, built to withstand a Category 3 hurricane.
After grazing Florida, Hurricane Katrina strengthened to a Category 4 and then a Category 5 storm on Aug. 28, 2005.
Many residents had already left the city as a precaution, and Nagin's mandatory-evacuation order convinced more to leave.
By then, Meffert had downloaded critical systems, such as financial management, and shipped them to the ACS data center in California.
Having those systems outside the disaster zone later allowed Meffert to instruct the data center operator to run payroll at a time when city government was otherwise barely functioning.
Meanwhile, Domke arranged to move the Web site from servers running in City Hall to a Dallas data center operated by Red Carpet Host, which offered the suite of Windows technologies the Web site required.
Having designed his own content management and portal software paid off here, he says.
On the Saturday before the storm, he was able to quickly copy his files to Red Carpet's servers by Internet file transfer, without worrying about software licensing issues.
"I was thinking of it as something that was precautionary only, knowing that the city's power grid was tenuous and anything could knock it out," Domke says.
Some of the more sophisticated Web applications weren't working at that pointanyone who was trying to apply for a business permit online during the storm would have been disappointedbecause the move separated them from the rest of the city's application infrastructure. But the basic Web presence would survive uninterrupted.
A week or so after the storm, help began to arrive. In addition to National Guard units and federal troops from units such as the Army's 82nd Airborne Division, emergency workers and freelance volunteers came with canoes and jet skis to help with the job of rescuing residents of the flooded areas from rooftops and isolated patches of high ground. Technical help arrived, too.
For instance, Minyard, the Unisys disaster recovery specialistand a former Green Beretcame to New Orleans on his own initiative and helped establish the city's Emergency Operations Center.
Minyard turned up bearing a portable emergency communications unit originally designed for the Port Authority of New York and New Jersey after the destruction of the World Trade Center towers; it contained everything needed to support multiple Internet, phone and wireless connections off a satellite link.
Called the Unisys Mobile CommHub, it packs everything, including a small satellite dish and a Cisco 3200 Mobile Access router, into a luggable crate.
"After hearing the pleas for help and knowing they had no communications, I was almost compelled to go," Minyard says.
But getting into the city at that time was a challenge in and of itself, and he made the last leg of the journey in a four-truck convoy with an armed escort.
"We literally drove in the middle of the night from Baton Rouge to New Orleans at about 100 miles per hour," he says.
"We went through a few checkpoints, and the basic question there was, 'Do you have enough ammunition?'"
The fear was that "the bad guys were better armed than the police, having broken into all the gun shops," he adds.
Meffert also got critical help from Port Authority of New York and New Jersey personnel, including information-technology people whom he calls "unsung heroes" for coming down on their own initiative, wanting to share what they had learned about disaster response after the Sept. 11, 2001, attacks.
"They gave us a lesson in Government Funding for Disaster Relief 101," he says.
With their advice, and technical support from Unisys, Meffert learned how to run an Emergency Operations Center sophisticated enough to coordinate the response to the immediate crisis, and how to begin the long, bureaucratic process of securing federal aid.
: Some decisions that paid off">
As the most intense phase of the crisis passed, a much longer-term financial and human crisis began in the devastated and deserted city. More than 120,000 homes had been damaged in the flooding, and the city needed to determine which ones could be salvaged.
One project Meffert had initiated before Katrina that turned into a significant asset during the crisis was geo-coding the city's property database, using geographic information systems technology from ESRI, a prominent mapping software specialist.
The project was partly intended to digitize Federal Emergency Management Agency flood maps for the city, and determining a property's elevation in comparison with the official flood levels was a critical element of determining a homeowner's eligibility for federal help and for a permit to rebuild.
Given geographic coordinates for every property, building inspectors could use Global Positioning System coordinates to identify a home even if all street address markings had been washed away in the flood.
Meanwhile, as residents wanting to rebuild or repair their homes flooded City Hall with permit requests, the city's Web-based permitting system helped make the volume of requests more manageable becasue fewer people had to wait in line at City Hall.
The permitting software vendor, Accela, a government software specialist, also helped the city develop and deploy a system to help building inspectors do their work quicker (see "Mobile Network" sidebar, p. 40).
Luckily, given the city's cash-strapped position, vendors with whom Meffert had established close relationships before the storm, such as Tropos, Intel, Microsoft and Dell, stepped in with donations.
Banks of phones and computers appeared in the Hyatt ballroom, with big-screen projections of maps of the situation throughout the city displayed on the walls.
Some of the computers came from FEMA, Meffert says, but Dell donated laptops worth about $500,000; Tropos donated 50 wireless mesh networking units while Intel paid for another 50, for a total value of $1.25 million.
Microsoft helped by settling a $700,000 bill from the city's payroll processing vendor and gave New Orleans a loan for another $1.3 million. Cisco threw in a maintenance contract worth about $400,000.
Meffert barely had to ask for this largesse: "In a weird way, I didn't have to."
The vendors knew him, and they saw the opportunity "to do the right thing and help themselves in the same breath" by proving the value of their technology in a highly public setting, he says.
But the city couldn't make a comeback solely on the basis of donations. Shortly after the storm, Meffert got roped into writing an emergency funding request for Karl Rove, whom President Bush had appointed to oversee the disaster response from FEMA and other agencies.
Meffert estimated that restoring city services would cost about $771 million over the next year.
The city expected to run a deficit of $358.6 million because of vanished tax revenue and increased expenses. An additional estimated $412.6 million would be needed to restore city buildings and capital equipment.
In the section of the capital budget under citywide services for public safety, Meffert included $12.6 million for the surveillance camera network, $9.8 million for the city's Wi-Fi mesh network and $18.5 million for the police digital radio network.
In a mid-November meeting with a group of FEMA officials, Meffert found himself defending the wireless networking component of his proposal.
Essentially, the FEMA guys wanted to make sure they would be paying to replace something that had been lost, and not for an upgrade.
Meffert argued that his recommendation would actually be more practical than trying to restore wired phone and networking equipment that in many cases was obsolete.
"We have to put in what makes sense," he said. Most of all, he wanted to maximize the effectiveness of the employees who would remain, following layoffs.
"They have to do the work of three or four people now," Meffert said.
The FEMA people were polite, but made no promises.
A couple of weeks later, Meffert and Nagin announced plans to blanket the city with Wi-Fi and open up the network for free public access.
The public network will use the same hardware as the one for city workers, so it required little additional investment.
Police and city workers would get secured, priority access to the wireless network, but unsecured connections would be offered to anyone who wanted to take advantage of them.
In part, this was to be offered as an economic stimulus, a way for returning businesses and their employees to get online even if wired Internet services were unavailable at their homes or offices.
"This is one more sign that we are rebuilding New Orleans into something better, something bigger, something wireless," Nagin proclaimed at a Nov. 30 press conference.
To date, wireless service has been a lifesaver for people trying to do business in the city, leading to odd sights like crowds of laptop-toting young professionals encamped at the tables outside a coffee shop after hours, because the cafe's wireless connection had been left on.
While the municipal Wi-Fi network is being tasked with doing double duty for public Internet access, the city retains the ability to shut down public access and dedicate the network to emergency communications, according to Chris Drake, the city's project manager for wireless infrastructure.
Meanwhile, he says, because the network is being used for routine tasks rather than only in an emergency, "that's like having a test of your emergency communications system every day."
By early 2006, Meffert was trying a new tack with the wireless network, meeting with EarthLink, Yahoo and Google about the possibility of taking it over.
The "cool part" of what the city offers to vendors like these is that in a community where so much has been destroyed, there's little reason not to try new things, Meffert points out: "I don't have Internet at my house, and neither does the mayor."
with political storms ">
On the morning of Jan. 5, as a City Council meeting is about to start, Meffert enters the council chambers carrying a pink can of Tab and wearing a beleaguered expression.
Outside, a group of protesters are reacting furiously to news reports quoting Meffert, in his deputy mayor role, saying the city plans to raze 2,500 homes whether the owners like it or not.
On the way inside, he accepts the praise of a local lawyer who has found the property-damage database on the Web an invaluable research tool.
Unfortunately, no Web site anywhere contains all the answers to the questions New Orleans residents are asking about the future.
In the coming election (postponed from February to April because of the logistics of reaching evacuated voters with absentee ballots), Nagin could easily be thrown out of office because of frustrations over the pace of the city's recovery.
Meffert was called here to discuss electrical inspections. But it turns out that City Council President Oliver Thomas is just back from visiting with the protesters, and their concerns are foremost in his mind.
When called to testify, Meffert gets thoroughly grilled. But he says the city is only trying to force the issue on about 120 homes that are particularly dangerous.
Most of what people are protesting is based on a misunderstanding, he insists: "Emotion without information breeds fear."
Meffert leaves the council meeting muttering, "That private sector is looking pretty good about now."
In a phone call a week later, Meffert is more philosophical.
"You know, this whole deputy mayor thing is sort of the CIO's dream and the CIO's nightmare at the same time," he says.
Other municipal CIOs complain to him that they can't get things done the way he can because of the opposition of other department heads.
In Meffert's case, a lot of the key department heads report to him. He doesn't have to worry about getting the utilities department to let him hang wireless equipment on the streetlights, for example.
On the other hand, he is in a "tight political spot," he admits.
"This is a hard, hard, hard place to work right now."
Meanwhile, as a technologist, he sees opportunities. At a time when so much needs to be rebuilt, why not make the systems integrated? Why not consolidate?
"All the things everyone says you ought to do, we're going to do," Meffert insists.
The experience of surviving Katrina also taught him a few things. "Everyone else who pitches the disaster recovery thing, that's all theory to them," Meffert says.
That's why the best assistance he got during the Katrina crisis came from the Port Authority of New York and New Jersey personnel, who came to New Orleans to share what they'd learned from their own catastrophe.
As Meffert vows: "I will return the favor that those guys did for me to the next city that has a major disaster."
information systems all play a role in New Orleans' recovery">
City of New Orleans Base Case
HEADQUARTERS: New Orleans City Hall, 1300 Perdido St., New Orleans, LA 70112
PHONE: (504) 658-4000
BUSINESS: Governing a major port city known for jazz and its nightlife before 80 percent of the city was flooded in the wake of Hurricane Katrina.
CHIEF INFORMATION OFFICER: Greg Meffert
FINANCIALS: The city is facing a projected deficit of $358.6 million because of vanished tax revenue and increased expenses.
CHALLENGE: Recover from one of the biggest natural disasters in U.S. history. Use technological efficiencies to compensate for the layoffs of more than 3,000 city employees, or about half the municipal workforce.