Stress Test To Avoid Stress

By Ericka Chickowski Print this article Print

Big IT projects sometimes go wrong in spectacular ways, with some common themes running through the disaster stories like fault lines.

Worst case scenarios happen. Anticipate and communicate, or they could happen to you.

Heartland Heartburn

Heartland Payment Systems started off 2009 with a bang, but not the kind it wanted.

The credit card processor suffered from a colossal security breach that reportedly exceeds 100 million records and affects accounts at hundreds of banks. An unknown party hid sniffer software in unallocated disk space on a server located within a section of infrastructure unprotected by encryption. Heartland’s CFO reported that the malware was so well camouflaged that forensics experts had a hard time finding it even after the Visa and MasterCard flagged the company with fraud alerts .

Details from the investigation will likely take years to unravel, but security experts speculate that it could have been installed from web-borne malware or perhaps even propagated by an insider armed with an infected USB device. Interestingly, the company had previously been certified as PCI compliant.

Heartland is suffering the consequences. Since it made the mandated breach announcement in January, its stock price has plummeted, and affected banks and consumers have filed dozens of class-action lawsuits. Competing card processors are aggressively courting Heartland customers. VISA and MasterCard revoked the company’s PCI compliance certification and are threatening to slap it with hundreds of thousands of dollars in fines. And CEO Robert Carr is now under fire for selling a large chunk of his shares in the company just before the breach announcement was made public. 

Lessons Learned: Compliance efforts may not always result in a secure risk posture. Security missteps can cost companies dearly.

Beijing Olympics Ticket Turmoil

If the Beijing Organizing Committee for the Olympic Games (BOCOG) IT team competed on the field, it wouldn’t even make it past the first heat in e-sales systems development.

During the run up to the Olympics, the Chinese ticketing system crashed not once but twice during two separate waves of domestic sales. The first time, in October 2007, Chinese authorities had released 1.8 million tickets to be sold online and at branches of the Bank of China (BOC). Within an hour the online ticketing system was unable to handle the heavy load of more than 8 million hits.

Designed to handle about 1 million hits per hour, the system crashed after selling a measly 43,000 tickets. Officials had to resort to offering the remainder by lottery while they worked with Ticketmaster to revamp their online sales portal. In the interim, BOCOG fired its director of ticketing.

Then in May 2008 the systems crashed again, this time in a bid to sell 1.38 million tickets. This go round the system was able to shop 300,000 tickets before going down.

The Beijing spin machine worked full force following the humiliation of both events, pointing to overwhelming demand for tickets as a signal of success. But ultimately the IT team failed to accurately project realistic traffic numbers and architect systems accordingly. The lack of planning cost one man his job and served up a splatter of egg on the face of countless others.

Lessons Learned: IT will fail again and again if it isn’t provided with accurate projections and numerical assumptions from the business side. 

This article was originally published on 2009-05-15
eWeek eWeek

Have the latest technology news and resources emailed to you everyday.