WhatBy Kim S. Nash | Posted 2006-08-02 Email Print
Know the Risk: Digital Transformation's Impact on Your Business-Critical Applications REGISTER >
When encountering legal or regulatory action, technology managers who fail to get corporate data fast or vouch for its completeness can cost their companies millions of dollars. Learn what happened to WestLB, an investment bank, when it had to exhume 650,s In Your E-Mail?">
What's In Your E-Mail?
In May 2005, Quinby changed lawyers, and her new legal team, headed by Kathleen Peratis, a partner at Outten & Golden in New York, expanded the ways they asked for WestLB's electronic evidence.
Peratis wanted WestLB to search for e-mail and Bloomberg messages from mailboxes of 19 current and former equities executives, human-resources representatives, bank managers and others, using more than 170 terms. These ranged from Quinby's name and initials, to employment-related words like "fire" and "bonus," to derogatory sexual slang.
"Outrageous," complained Joel Cohen, another lawyer for WestLB, to Judge Pauley. The request would cost the bank "hundreds of hours of work," according to a transcript of a hearing in June 2005. It took the bank another month and several more hearings to get the request narrowed, to 17 people and just a few search terms. For most mailboxes, the key words were "Claudia," "Quinby" and her initials. But for others, there were extra terms, such as "lawsuit," "fire," "axe" and "complain."
But even this tapered search proved challenging because of the way backup tapes work.
Of the three places where WestLB stored e-mailLotus Notes, AXS-One and backup tapesthe tapes were the most complete source of information. They captured messages companywide and at least some messages that employees had later deleted off the active servers.
For many companies, backup tapes can be a savior and a nemesis. At $20 to $50 each, tapes are cheap. And unlike a hard disk, tapes are sturdy enough to travel well to an off-site storage facility. Backup tapes work well in disaster recovery because they hold a snapshot of a company's systems at a given point in time. Restore the tapes, and you can re-create the downed system. New Orleans-based Oreck Corp. relied on tapes to get running again after Hurricane Katrina last year.
But tapes are also the most expensive source from which to extract e-mail because of the specialized knowledge involved to pinpoint the kind of specific data demanded in an audit or lawsuit.
To restore a tape, it must be run on the same brand and version of the software and server from which it was originally made. To search a tape, Kroll technicians first copy it to protect the data, explains Dave Schultz, a senior legal consultant at Kroll. He declined to talk specifically about Kroll's work at WestLB, but described his company's general practices.
Kroll then uncompresses the data and restores it onto a server in a format that humans can read. Each of the mailboxes in question would then be searched electronically for e-mail with subject lines or body text that contained the approved keywords.
As Jessica Grossman, another lawyer for WestLB, explained to the judge, she and her team would type a search word and get back a long list of documents containing that term. They would then have to open each document to find the word, highlighted in blue.
"If the word came up on page 38 of a 70-page document, we would have to sit and go from page to page to find where the word came up. It was hard on your eyes sitting on the computer there," she said. "We were working around the clock."
Backup tapes are not indexed, so data must be read in sequence. Hundreds of thousands of e-mails may sit on a tape, each with multiple threads that accumulate as they are sent back and forth with additions or links to different people. Then, lawyers must check every message that's recovered, to judge whether it's pertinent or private attorney-client material.
"A backup tape of a server is like a bunch of text dumped into a 55-gallon drum," says David Duryea, a principal with Endeavor3 LLC, a forensics expert in Beachwood, Ohio. He advises his clients never to use e-mail to conduct transactional business, such as customer service, because the contents are so hard to track.
In mid-July last year, Kroll got overwhelmed by the sheer volume of data on the tapes, WestLB told the court. Kroll's estimate of three to six weeks to complete the job stretched to 3 1/2 months.
To try to make the court's deadline, which was pushed from July to August to September, WestLB brought in HP, its outsourcer, to help Kroll. HP copied the contents of 16 e-mail boxes from the Lotus Notes server onto CDs, according to Kevin Faulkner, a risk management consultant working for Quinby's side. That meant fewer backup tapes for Kroll to process.
Faulkner maintains the bank should have been able to handle that job on its own. But "neither WestLB nor HP is experienced in or purports to have expertise in electronic discovery issues," said Michael Waxenberg, director of business infrastructure management in the bank's technology group, in an affidavit. HP says it does not comment on customers.
Also a problem was getting usable e-mail from the bank's AXS-One system, Waxenberg told the judge in an August 2005 hearing. When trying to get search results out of AXS-One's proprietary format and into something more common, such as a Lotus Notes file, most attempts to export files from AXS-One crashed, inexplicably, at around 150 messages, he said. Not helpful when result sets contained as many as 10,200 messages. Kroll was unfamiliar with AXS-One data formats, and the bank was reluctant to get help from AXS-One, Waxenberg said. "I have a rather acrimonious relationship with them because we're in the process of decommissioning their product," he told the judge. WestLB didn't have a support agreement with AXS-One, he added, so any help from the vendor would have meant paying it a consulting fee. AXS-One CEO Bill Lyons says the company does not comment on specific customer installations.
Meanwhile, Bloomberg turned over at least 75,000 pages of WestLB's instant messages, in Microsoft Word format, on CD-ROMs.
"This is the most enormous e-mail search the bank has conducted in a litigation," Groman Darringer told the judge in that same August hearing.
By October, Kroll's searches had produced 649,517 electronic pages of e-mail, human-resources records and other evidence from 75,281 different documents, according to an affidavit by Kroll project manager Lori Carey. Kroll had billed WestLB for more than $380,000, including a 25% premium for expedited processing. HP estimated the cost for its help at another $100,000, Faulkner said.
Three-quarters of the pages and 86% of documents came from the backup tapes, according to Carey. Being reliant on tapes cost WestLB hundreds of thousands of extra dollars, Faulkner estimated.
That is partly because of the way data is stored on tapes, but also because, in Faulkner's view, Kroll should have bypassed the tapes to take e-mails directly off Lotus Notes for about $10,000, a fraction of what WestLB was charged. Kroll's Schultz declined to comment on WestLB but says Kroll's pricing is "pretty much standard."
And the searches, Groman Darringer acknowledged in court, were "not perfect." For example, they turned up lots of false positives, or irrelevant documentsa "significant portion" of the first 220,000 pages produced, according to WestLB lawyer Cohen.
Kroll's "ineffective" search techniques produced these extra pages, Faulkner claims. For example, the search term "CQ," Quinby's initials, delivered documents with variations of the word "acquire," which is used frequently in a bank. "You can't just look at keywords," Faulkner says. "You have to look at the meaning behind them." In an affidavit, Faulkner said Kroll could have structured the query by using syntax to avoid turning up words that contain "CQ"by typing spaces before and after the letters.
Schultz says his company's use of search techniques, including tiered searches to narrow results, depends on "the time and money available."
On each set of search results, WestLB's legal team then ran another search for words like "attorney," to identify documents that might be private attorney-client privileged material and therefore not to be presented to the other side. Lawyers then had to review these pages manually to glean context and judge whether each one qualified as privileged. Despite the scrutiny, at least 100,000 pages of private material slipped by and went to Quinby's lawyers by mistake, according to a hearing in February 2006, during which WestLB argued for the return of some of the documents.
For example, one e-mail message filed in court says WestLB was eager to settle with Quinby and avoid a lawsuit, "given that we already have 2 other lawsuits already filed with the EEOC [Equal Employment Opportunity Commission] about gender discrimination." That message was written in 2003 by the bank's former head of human resources, Betsy Austin.
The bank had to fight in court to get those pages back, explaining to the judge how "onerous" the multilayered electronic search and subsequent manual review were.
"It was like finding needles in a haystack," Groman Darringer said in court. "We had cadres of lawyers having to go through this production. And there were so many documents, a review could only be done once" to allow WestLB to meet the court's deadline.