Lessons in Chaos

 
 
By Mel Duvall  |  Posted 2007-04-05
 
 
 

As an airline industry veteran, Charles "Duffy" Mees has had to weather his share of storms over the years. Yet nothing, says the vice president and chief information officer of JetBlue Airways, could have prepared him for the series of events that culminated in a virtual shutdown of the airline's operations following a Valentine's Day storm this February.

Barely four months into his post as the CIO of the Forest Hills, N.Y.-based discount airline, Mees watched with increasing dread as a combination of deteriorating weather and poor management decisions led to chaos on the tarmac at John F. Kennedy International Airport in New York and other major centers in the Northeast, including Washington, D.C., and Newark, N.J. By the end of the day, passengers, some of whom had been left stranded on planes for as long as 11 hours, were demanding management's heads.

Mees, who was still getting a handle on the various systems used to run the airline's operations after concentrating on completing an SAP enterprise resource planning installation at JetBlue, watched as one by one, the airline's primary information and communications systems buckled in the following hours and days under extreme loads. Mees joined JetBlue as CIO in November 2006, replacing Todd Thompson, who became CIO for Starwood Hotels. Previously, Mees had served in senior I.T. roles with Reno Air and Independence Air.

The resulting public relations nightmare dealt JetBlue a tough lesson in what can happen when you don't plan for disaster. Prior to the event, JetBlue was a much admired success story, having grown since its first flights in 2000 into an airline with $2.4 billion in revenue by the end of 2006, operating 500 daily flights to 50 cities. Its sudden fall back to Earth offers CIOs of other fast-growing small and midsize companies a stark lesson in what can go wrong and how to prepare for the worst.

"In the heat of battle at any rapidly growing company, you're always trying to address your most immediate needs," Mees says. "But you've got to continually remind yourself that you have to take a step back and look at the things that aren't right in front of you—find out what the tipping points are—before they can impact you."

Over the three days following the meltdown at JFK, Mees and his information-technology team would barely manage an hour's sleep each night as they scrambled to support beleaguered employees at airports. Despite the surrounding turmoil, the JetBlue technology team put in a heroic effort to get operations back on track, pushing systems to their limit and creating new databases, tools and applications on the fly to solve problems as they arose.

's Embrace">

Winter's Embrace

The weather front that led up to the Valentine's Day storm had already been causing trouble in the Midwest for several days before it reached the East Coast. But while it primarily delivered blowing snow and frigid temperatures in cities like Chicago, the storm had a different cargo by the time it reached New York. Freezing rain and sleet slammed the mid-Atlantic region, causing havoc on major highways and virtually shutting down airports.

While most other airlines cancelled dozens of flights in preparation for the storm, JetBlue management opted to wait it out. The airline's policy is do to whatever it can to ensure a flight is completed, even if it means waiting for several hours. According to chief executive officer David Neeleman, the airline sent outbound flights to the runway at JFK at about 8 a.m., to be ready to take off as soon as the weather let up, while incoming flights arrived and filled up the gates. But instead of improving, the freezing rain and sleet continued. Under federal aviation guidelines, planes cannot take off in ice pellet conditions. Soon, planes and equipment were literally freezing to the tarmac.

By 3 p.m., the airline gave up hope of getting the planes sitting on the runway off the ground and began calling in buses to bring passengers back to the terminal. But by then, the damage was done. Airport terminals, particularly the JFK hub, were filled with passengers still expecting to get on flights and they were now being joined by hundreds of infuriated passengers getting off planes. "Things spiraled out of control. We did a horrible job," said Neeleman in a conference call early the next week. "We got ourselves into a situation where we were doing rolling cancellations instead of a massive cancellation. Communications broke down, we weren't able to reach out to passengers and they continued to arrive at the airports ... it had a cascading effect."

In fact, the impact of the Valentine's Day storm would last for almost a week as JetBlue attempted to accommodate passengers on planes that were already nearly full. In response, Neeleman introduced a JetBlue Customer Bill of Rights offering various forms of compensation to customers whose flights have been cancelled or are left sitting too long on planes. The direct financial impact was estimated by Neeleman to be as high as $30 million, although he says it's the blow to the company's reputation that stings the most.

During his conference call with investors, Neeleman made a point of stating that JetBlue's computer systems were not to blame for the meltdown. "It's important to note our software providers didn't fail," he said. While Mees agrees with that assessment, he also admits that there were a number of shortcomings in the company's information systems. In some instances, such as the reservation system, the airline was not able to expand the system enough to meet the extreme customer call volume, while in other instances, such as keeping track of off-duty flight crews and lost baggage, there simply weren't systems in place.

As the seriousness of the situation began to unfold on Valentine's Day, managers with JetBlue's Salt Lake City-based reservation office began calling in off-duty agents to assist with the expected high volume of calls. JetBlue reservation agents primarily work from their homes, using an Internet-based communications system to tap into the company's Navitaire Open Skies reservation system. Navitaire, headquartered in Minneapolis, hosts the reservation system for JetBlue as well as about a dozen other discount airlines.

As passengers were bumped off planes or arrived at JFK and other East Coast airports to find their JetBlue flights had been cancelled, they had only one option to rebook their flights: call the JetBlue reservation office. JetBlue does not offer its customers the option to rebook their flights via its Web site, nor can passengers rebook using airport kiosks. According to Mees, the option of rebooking via JetBlue.com is something the company began working on in December 2006, before the February disruption. But the new feature was not yet ready to be rolled out. As a result, the Salt Lake City reservation agents were suddenly deluged with calls from irate passengers looking to get on another flight or find out what compensation was available.

The Navitaire reservation system was configured for JetBlue to only accommodate up to 650 agents at one time, a number that more than met its requirements under normal circumstances. As customers' wait times on phones ballooned past an hour—if they could get through at all—Mees put an urgent call through to Navitaire to see if anything could be done to increase the number of agents using the system. Navitaire was able to boost the system to accommodate up to 950 agents at one time, but then it hit a wall. More agents could not be added without impacting system performance. Mees says Navitaire is working on an upgrade to the system that will allow it to expand to substantially higher numbers, but for now it has reached its limit.

Baseline contacted Navitaire for comment, but it referred questions about timing and capacity issues back to JetBlue.

Even with the number of agents that could be accommodated expanded to 950, JetBlue was having difficulty finding enough bodies to staff the phones. It employs a total of about 1,500 agents. Off-duty crews and airport personnel volunteered to staff phones, but they were not trained in how to use the system. Such system cross-training is not common at most airlines, but it's something Neeleman vows will become common practice at JetBlue in the future. He says management and certain staff segments will be cross-trained on such functions as reservations and crew scheduling to bolster ranks during an emergency.

As passengers struggled to get through to reservations, their bags piled up in huge mounds at airports, particularly at the airline's hub at JFK. Surprisingly, JetBlue did not have a computerized system in place for recording and tracking lost bags. In 2003, it entered into an agreement to purchase a system from Lufthansa called BagScan, but the system was never implemented, Mees says. "We didn't prioritize it—probably because we were so focused on the SAP project," he admits. JetBlue was growing quickly and needed the increased capabilities in the SAP ERP system to handle its human-resources functions.

JetBlue has a data warehouse that stores reservation and check-in information, such as the number of bags checked in by a passenger and the bag tag identification numbers. What was missing was an add-on component to record which bags had not been picked up and their location. There was no way for a passenger agent, for example, to look up by computer if a lost bag for a particular passenger was among the heap of unclaimed bags in New York. Not having this functionality had never been a big problem in the past, Mees says. If there were bags left over at the end of a flight, airport personnel figured out ownership by looking up a passenger record. Because JetBlue rarely cancels flights, the process had been manageable.

Not this time. "There were so many bags, they went from floor to ceiling—and where there weren't bags, there were people," Mees says. A technology team (JetBlue has 193 people in its I.T. department) was dispatched to the airport to help out with the effort. They ended up hauling most of the bags to an offsite location where the bags could be sorted and identified. Over three days, programmers cobbled together an application using a Microsoft SQL database and handheld devices from Symbol Technologies that could scan a bag tag and identify the passenger. Agents could then access the database to provide passengers with information on the location of their lost luggage.

Meanwhile, in flight operations, managers were attempting to sort out a number of other headaches. In addition to the Navitaire reservation system, JetBlue uses several applications from Sabre Airline Solutions of Southlake, Texas, as part of its core operations infrastructure. The Sabre Flight Control Suite provides the airline with applications to manage, schedule and track its planes and crews, while Sabre's Dispatch Manager application is used to develop the actual flight plans. Sabre's FliteTrac application, part of the Flight Control Suite, interfaces with the Navitaire reservation system and provides managers with real-time information on factors such as flight status, fuel information, passenger lists, and the original, revised, estimated and actual arrival times. The Sabre CrewTrac application, also part of the Flight Control Suite, tracks crew assignments, ensures legal requirements are met, and provides pilots and flight attendants with access to their schedules via a secure Web portal.

JetBlue also utilizes a sophisticated system from Navitaire called SkySolver to help figure out the best way to emerge from flight disruptions. SkySolver allows operations planners to plug in a number of scenarios, such as canceling flights or redeploying planes and crews, to figure out which actions will get operations back on track the quickest while minimizing passenger disruptions. It might, for example, recommend a plane traveling from Burlington, Vt., to Orlando, Fla., with a stopover in New York, fly directly to Orlando if that is the final destination of most passengers. JetBlue has used SkySolver in the past to help it during major disruptions, including the 2004 hurricane season.

Mees says early on during the disruptions on Valentine's Day, planners worked out a number of scenarios using SkySolver to get their operations back on track. However, when they attempted to transfer the solutions—or "solvers," as they are called—from SkySolver into the company's Sabre applications to produce flight plans that could be quickly acted upon, SkySolver was unable to transfer the information into Sabre. According to Mees, Navitaire immediately tackled the problem and was able to come up with a fix in a matter of hours, but by the time the problem was fixed, "the damage was already done." Gates were full, flights could not be cancelled in an orderly fashion, and the airline virtually reached gridlock.

Mees says he doesn't know why the information could not be transferred or why the glitch was not discovered beforehand. Navitaire would not provide a response to Baseline's questions. For now, Mees says he has other priorities to work on and was satisfied with Navitaire's quick response.

Even if JetBlue had been able to work out a game plan to bring a quick end to the disruption, there was no guarantee it could have gotten flight crews to the redirected planes. In the days following Valentine's Day, JetBlue struggled to keep an accurate account of its pilots and flight crews. Normally, during disruptions, off-duty crews call into headquarters to give their location and availability to work. But with the sheer number of flights affected—more than 1,000 were cancelled over the three-day weekend for Presidents Day—phone lines were busy, calls couldn't get through, and information had to be cobbled together with pen and paper. JetBlue did not have a database system to keep track of off-duty crews in such a situation.

Once again, the technology team created a simple, centralized database over a 24-hour period to keep track of crew locations and their contact information, using a Microsoft SQL server database and .NET tools. Mees says he will keep that system for the interim, but the plan is to develop a more full-featured system that might allow pilots and crews, for example, to plug in their location and availability via the Internet or a handheld device.

It's just one of a number of initiatives that have suddenly vaulted to the top of Mees' priority list. Others include:

  • By early April, work with Navitaire to add a feature to JetBlue.com that will allow passengers to rebook cancelled flights without having to call reservations.

  • Install up to 150 thin-client computers, with Internet connections, behind security at JFK and other hubs, to allow passengers to rebook flights at the airport using JetBlue.com.

  • Work with Navitaire to double the number of agents that can be accommodated on the Open Skies reservation system as soon as possible.

  • Enhance the new lost-bag tracking system so it can become a core application.

  • Implement new systems that will allow JetBlue to notify passengers by e-mail, phone or its Web site of cancelled or changed flights as soon as possible to prevent lines at airports.

  • Institute cross-training of staff on reservation, flight and crew scheduling applications.

In other words, bolster JetBlue's information infrastructure so that it can offer many of the features now being offered by its major airline competitors.

Lessons in Chaos

Analysts say JetBlue is far from alone in suffering an operational meltdown. In fact, there were a number of other passenger horror stories at other airlines this past winter.

How JetBlue responds to its failings, however, will determine whether passengers return.

"Until this occurred, I don't think JetBlue had a test of their systems," says Dean E. Headley, associate professor of marketing at Wichita State University and co-author of the Airline Quality Rating, which ranks airlines according to criteria such as online performance, mishandled baggage and customer complaints. JetBlue was the number-one-rated carrier in the ranking for 2003, 2004 and 2005 (the 2006 rating is expected to be released in early April).

Vendors and analysts alike also point to a number of new technologies in the wings that may help airlines like JetBlue. Sabre launched a new system last fall, for example, that allows airlines to figure out the best alternative to a cancelled flight for an individual passenger and automatically rebooks it. "An airline can now re-accommodate all the passengers from a fully loaded jumbo jet in five minutes—a process that in the past took hours," says Ilia Kostov, vice president of product management and marketing at Sabre Airline Solutions.

Airlines are also capitalizing on new business intelligence applications to make better use of centralized passenger databases. Lydia Pearce, a senior partner in the travel industry practice with Teradata, a provider of data warehouse systems, is working with a large U.S.-based airline (which could not be identified) that installed a centralized database to capture all passenger information, from frequent flyer numbers through to reservation information and actual flight details. Using that database, the airline now runs business intelligence queries to determine, for example, how many flight disruptions, cancellations or delayed arrivals its "gold" level members experience in a given month. That way, the airline can proactively reach out to customers who may have been inconvenienced and ensure their continued business.

"They are able to automatically generate an apology and offer some sort of compensation, and then watch that passenger's record to see if it had an effect," Pearce says. "Capturing the impact of the action you took is just as important as the action."

IBM is also working with a number of airlines, including JetBlue, to install more sophisticated kiosks at airports. With these kiosks, says Bruce Speechley, a partner in IBM's travel and transportation practice, airlines may be able to automatically present passengers on cancelled flights with rebooking options, avoiding calls to reservation offices. Some airlines, Speechley says, are experimenting with systems that will present a passenger with a rebooking option, and if they choose not to accept it, offer three or four alternatives.

No matter what technologies JetBlue does decide to put in place, it won't matter if management makes the wrong calls.

For his part, Neeleman vows to learn from the event and win back customer loyalty. That promise was quickly put to the test when the East Coast suffered other major weather disruptions on Feb. 26 and March 16. This time, JetBlue immediately cancelled flights systemwide and was able to get its operations mostly back to normal the following day.

As a new CIO, Mees says the experience taught him more in a few days about the company's systems and employees than he would have learned in months.

"It helped me gain a higher respect for my team, and get to know each of them and their skills better," he says. "I think it was also beneficial for the technology staff in general. They hadn't spent a lot of time out at the airport. Now, they've seen how the systems are being used under extreme conditions—that's tremendously valuable."

JetBlue Base Case

Headquarters: 118-29 Queens Blvd., Forest Hills, NY 11375
Phone Number: (718) 286-7900
Business: A fast-growing low-cost passenger airline, operating approximately 500 daily flights and serving 50 destinations.
Chairman, CEO: David Neeleman
CIO: Charles Mees
Financials in 2006: $2.4 billion in revenue; $1 million loss.
Challenge: Win back customer loyalty in wake of major disruptions in service from winter storms in February.

BASELINE GOALS

  • At least double the number of agents who can simultaneously use its reservation system, from 650 in early February to more than 1,300 in six months.
  • Fast-track a project to add the ability to rebook a seat via the company's Web site, completing the work in one month instead of three.
  • Limit the time passengers can be held on planes to fi ve hours before they must be deplaned, from previous system that left the decision to the discretion of pilots.
  • Refine a new database system to track lost baggage so that it can be a core JetBlue application, instead of previous system that relied on manual processes.