Consumer Products: When Software Bugs Bite

 
 
By Deborah Gage  |  Posted 2003-01-16
 
 
 

Maurice Bailey's Miele G885 SC dishwasher cleans dishes almost as well as a human being. Its 10 separate programs control the washing and drying of fine crystal and crusty pans. Its electronic controls warn owners if the drain is blocked. It also carefully regulates both the temperature and the consumption of water, something humans often neglect to do.

PDF Download Mechanically, the dishwasher has never failed. But it was rendered useless after a power outage. Its software got knocked out. Bailey pored over the manual and then spent half an hour on the phone with customer service. Finally, Miele sent a technician to his house to reboot Bailey's dishwasher.

"Software is so pervasive," says Bailey, a general partner in the Bailey Group, which provides consulting services to technology companies. "This is a great product when it works. But I think this dishwasher is symptomatic of issues we are not aware of."

Miele declines comment. But many companies are exploring the potential of software to improve products by making them more durable. Software replaces knobs that break and mechanical parts that wear out, and it allows customers to fix and add features without buying a new machine. Software updates can be delivered to customers over the Web.

However, software lags hardware as a reliable component of products. And, analysts say, unless companies do a better job of testing and developing software, problems like Bailey's will likely get worse. Several analyst firms—including Patricia Seybold Group, ZapThink, and Venture Development Corp.—blame the technology industry itself, which is loosely regulated and tends to rush products to market to try to gain market share.

Consider BMW and its luxury 745i sedan. First released in Europe in November of 2001, the car contains around 70 microprocessors. Its most striking feature, iDrive, is what Car and Driver magazine classifies as a "miracle knob." This single element of the dashboard is designed, through a computerized console, to replace more than 200 buttons that control everything from the position of seats to aspects of the navigation of the car itself to climate, communications and entertainment systems.

The iDrive is powered in part by the stripped down version of Microsoft's operating system for personal computers known as Windows CE. Theoretically, Beemer drivers can adjust anything, move forward and not take their eyes off the road. But that assumes that iDrive is working.
A Worldwide Recall

Gary Conley, a retired executive in Silicon Valley, is now driving his second BMW 745i. But it's not because the first one worked that well.

His first car had so many problems that BMW bought it back last July. This buy-back followed a worldwide recall in May of 15,000 7-series cars, and a second recall in July of 286 vehicles by BMW Korea. The Korea Times cited a software bug in the electronic management unit of the vehicle's fuel pump that could make the engine stall. BMW spokesman Gordon Keil says certain cars stalled if the fuel tank was below 1/3, although this was not a problem that Conley ever reported.

Conley says his second car works better than his first one. For example, when it creeps along the road, it does not automatically brake without using the brake lights, as the first one was prone to do. But it still has intermittent problems that BMW can't reproduce or fix. The voice activation system sometimes fails, the transmission slips, the phone may fail to power up, and the iDrive settings have spontaneously disappeared and switched to metric units.

Indeed, Conley became so frustrated with BMW that he posted videos of his errant car, along with his most recent repair records, on the Web. (Click here for a collection of several of these links.) Out of 24 problems cited by Conley in December, his dealer was able to find and fix only three of them, despite help from U.S. headquarters in New Jersey.

"BMW tried to do too many things at once with this car, and they underestimated the software problem," says Conley, who built test equipment for semiconductors as the CEO of EPRO Corp., which sold to Credence Corp. of Fremont, CA, in 1995. "Only two-thirds of hardware has been unleashed by software. There are so many predecessors and dependencies within software that it's like spaghetti-ware. It's not that easy to get all these little components to plug and play."

Conley's situation may be unusual, but not unique. "About a month after I took the car, my iDrive system totally failed," says Ron Burke, a partner in the law firm of Brand Brand & Burke in New York City. "This left me able to drive the car, but unable to operate the radio, telephone or navigation system. BMW explained that it would take a long time to fix it because only a few people were qualified to address the problem. But they did fix it and it's worked ever since."

BMW's Keil says the company has sold over 22,000 of the 745's—a 64% increase over the previous 7-series—and has many happy customers. He says that BMW will work with Conley until he, too, is happy, although he questions whether some of Conley's problems could be solved if Conley had better instruction on how to use the car. For example, Keil says, the car's instrument clusters will reset themselves if the battery is low. Keil also says BMW has done "extensive testing" of the 745's.

Auto industry expert Dennis Virag, president of the Automotive Consulting Group Inc., says the problem is not customer ignorance, but industry carelessness. In the race to add glitzy amenities like navigation, Virag says, auto manufacturers are contracting out the development of immature and faulty software. "The auto industry is highly regulated, and these are not mission-critical systems," he says. "But companies like Microsoft can't do to the auto industry what they did to the PC industry. You can't play Russian Roulette every time you stick the key into the ignition."
Eliminating Quality Assurance

Some vendors agree software must improve. IBM worries about the quality of its own software, especially as its customers begin doing business over the Internet and interact with software and services that IBM has not developed. IBM recently acquired the tools vendor Rational Software, and in May, signed a global license agreement with Parasoft, a vendor in Monrovia, Calif., whose tools prevent software errors by automating the way software is developed.

Software is no different than any other manufactured product, says Parasoft CEO Adam Kolawa, and companies waste resources and lower quality by treating software as a work of art.

Parasoft's own developers follow a strict routine and are required to turn in their code for testing every night by 7. Kolawa says he eliminated Quality Assurance (QA) testing—which traditionally occurs after software is developed—years ago, because code would bounce between developers and testers and no one would take responsibility for problems. "Eliminating QA worked marvels," he says. "It enforces a culture of ownership. It enforces discipline."

But ABN AMRO Services Company Inc., which works with both IBM and Parasoft to help integrate systems for online banking, says automated testing tools don't guarantee good software. They won't work unless companies invest in planning and training, says VP John Schmuck—companies without automated, repeatable processes can't automate testing. Furthermore, no one tool covers the entire software development process, so tools must be integrated. And some parts of the process still have no tools at all.

The automotive industry is working extensively with software modeling, says Chris Lanfear, a senior analyst with Venture Development Corp. This lets them generate code from graphic models of systems rather than handwriting millions of lines of code. But he says the industry is still coming to grips with the fact that it's in the software business and must learn to manage the entire software development process.

For instance, General Motors is working on "drive-by-wire" cars that use a videogame-like controller—called an x-drive—linked to sophisticated software and hardware systems in a new kind of chassis with no mechanical parts. But the company's chief technology officer is not sure software companies are ready to provide either the software or the support at the level of reliability required by cars.

Auto manufacturers are legally bound to provide parts and service support for 10 years after each car is first sold. As GM CTO Tony Scott asked last year of attendees at Internet World in New York, "Is anyone here still running software from 10 years ago and could they find support for it?"

It "just won't be an option for us" to tell customers "to hit control-alt-delete to reboot the car," Scott said (See also, "Grand Test Auto," Baseline, Issue 012, Nov. 2002, p. 42).When Improving Software">
What You Should Do When Improving Software

What You Should Do When Improving Software
  • Plan ahead
    Think through your testing process.
    Test code's behavior under both expected and unexpected conditions.
  • Test early
    The earlier the better.
    Appeal to engineers who enjoy breaking things.
  • Test manually; Then, automatically
    Before code settles down, test manually.
    Later, have engineers use automated tests, before turning code in.
  • Don't test all code at once
    Clean up old code as each file gets run.
    Fifty percent to 80% of your files should never have to be touched.
    Source: ABN Amro Services Co.Quirks">
    Online Clips of BMW Quirks

    The following clips—all in Windows Media format—are from the www.bimmer.org Web site.

    Roll over Beethoven (http://www.h2go.org/745proof/rollover%20beethoven.WMV)
    Missing in Action (http://www.h2go.org/745proof/CD%20missing%20in%20action.WMV)
    Where am I? (http://www.h2go.org/745proof/no%20map.WMV)
    Crazy Trunk (http://www.h2go.org/745proof/745%20trunk.wmv)
    Spitn' Key (http://www.h2go.org/745proof/key%20spit.wmv)
    Lost Voice & Phone (http://www.h2go.org/745proof/no%20voice%20or%20phone.wmv)
    Phone Dead (http://www.h2go.org/745proof/phone%20dead.wmv)
    Metric Radio Options (http://www.h2go.org/745proof/metric%20radio%20edit.wmv)
    Transmission (http://www.h2go.org/745proof/translip%20maybe.wmv)
    Radio & Slip (http://www.h2go.org/745proof/radio%20and%20slip.wmv)

    The following images also come from the bimmer.org site.
    Dec 0 (second service on car) (http://www.h2go.org/trialhtml/02Dec02%20service.htm)
    CD or FM photo (http://www.h2go.org/cd%20or%20fm.jpg) ("shows CD on right instead of FM stations...when turned car off, CD was playing...came back to find radio (FM 106.5) playing")