Quashing a Bug Before It Alights

Security is again blowing a hole into Microsoft’s relationships with its customers—both individual and corporate.

The Blaster worm and its siblings took down many corporate networks in late August, and cost up to $1 billion in lost productivity and overtime—even though patches that could have prevented the problem were available in July.

But even for companies unscathed by this latest round of maleficent software, Microsoft’s security holes are having a significant business impact. For some companies, the prescribed cure to Microsoft software bugs may be just as bad as the disease.

“Patch management” is a euphemism for unnecessary pain. Many customers just don’t have the resources to devote to testing the impact of every new patch on their existing applications—and those that do frequently find that patches break software that they depend on to run their business.

Deploying even a single patch in panic mode can be costly. Citigroup, for example, had dozens of technical employees at each of its business units this summer working almost exclusively on deploying the latest bug fixes for more than a week, according to staff working on the problem. And this was before the Blaster burst into general awareness in August.

Citi won’t comment officially, except to note it didn’t suffer any security breaches. But the logistics of applying collections of patches to every single desktop computer and file server in the company’s inventory—and at least four different versions of the Windows operating system across all of them—is a gargantuan challenge.

According to one Citi network technician, patching was slowed down by differences in the distribution of Microsoft’s service packs for Windows 2000 across the network. The patch for Windows 2000 required that Service Pack 3 for that operating system be installed, for instance. On its end, Citi lacked a consistent way to test whether patches had been applied successfully. That’s a problem with installing patches on remote servers and desktops.

There were other stumbling blocks. “I had some [systems] I couldn’t log onto [or] didn’t have administrative rights to,” Citi’s technician told me. Rather than distributing the patches electronically, Citi’s technical “ground-pounders” had to go out on foot and get physical access to desktop computers.

Citi isn’t alone in such struggles. Bill Anderson, lead product manager for Microsoft’s enterprise management division, says these sorts of problems are common to many of its customers. “Citi is probably pretty typical for a large enterprise customer,” he says. Large enterprises “often don’t have a centralized top-down approach for things like patch management, or security in general.”

Microsoft’s partial solution to software update woes is the Software Update Services “feature pack.” This is a set of tools for its System Management Server (SMS), which package updates and automatically deploys them to systems that need them. The functionality will be an integrated part of the next version of SMS, which should be commercially available this fall.

But patching en masse—even automated patching— isn’t always the best answer. “You may look at [a new security hole] and say, ‘I can block these ports and not have to patch right away,'” says Anderson. The only way to know what course is best is by having a good handle on what you have installed—and good documentation of how your applications work.

Unfortunately, keeping your own house in order doesn’t guarantee the next Microsoft security loophole won’t affect you.

A big chunk of the downtime at companies hit by Blaster and its ilk was caused by computers owned by consumers. That’s a problem that can only be addressed by Microsoft—and corporate customers should hold Microsoft’s feet to the fire to do so.