BofA's Direct-Deposit Debacle

By David F. Carr Print this article Print

Payday is a great day, usually. But for thousands of Bank of America customers, it was the worst, as their electronic deposits vanished.

Bank of America customers in California, Nevada and Arizona got a rude shock on the weekend of March 15 when they tried to enjoy their paychecks.

They couldn't.

PDF Download Payroll, Social Security and other electronic payments scheduled to be deposited into their accounts on Thursday night disappeared from their accounts on Friday. Checks bounced. Withdrawals were blocked, because of allegedly insufficient funds. Confidence in banking automation was shaken.

In a rare lapse for the direct-deposit system, Bank of America lost track of money that should have been credited to tens of thousands of customer accounts, and took days to straighten out the problems.

In the confusion that followed the incident, representatives of the bank initially blamed the national Automated Clearing House (ACH) system. But that system worked. The bank, instead, soon had to admit to an internal data processing problem—not a software failure, but something it would term an "operating error" at the bank's data center in Nevada.

That error resulted in a batch of ACH transactions not getting processed. "Memo" transactions were entered, making customers believe the transactions had been completed. But such notations actually are temporary credits to an account for a scheduled payment. If the payments aren't actually deposited, they disappear.

"During the day when a transaction occurs, we do a temporary posting to the customer's account that is immediately available. Then, at the end of the day when we do our batch processing, we actually run it through the transaction process for permanent posting," says Brad Russell, public relations manager for technology and operations at Bank of America. "Due to the operating error, a number of transactions that had been memo-posted were not transferred to the permanent system."

Making matters worse, it took the company most of the weekend—a period of about 36 hours—to restore the amounts due to the accounts affected.

Fortunately, the transactions didn't completely disappear. Even though Bank of America's systems had already acknowledged the receipt of the transactions, they were backed up at multiple points in the automated clearinghouse network. While the clearinghouse system is usually referred to as a single network, it is in fact a network with four separate hubs. Each hub has multiple levels of backups and redundancies, according to NACHA, the National Automated Clearing House Association. ACH transactions are typically passed along as batches of electronic documents, pulled down from the clearinghouse and acknowledged by the member institutions. If a transaction is dropped, it can be re-sent from any point in the system.

Bank of America, like other financial institutions, also has its own redundant and fail-over systems. Using storage management software similar to that from EMC, Hitachi or Computer Associates, database images can be replicated in "snapshot" form, and incrementally refreshed, on standby or backup storage servers. That data can be brought on line almost instantly from any known good point, or it can be copied back over damaged data.

In fact, Bank of America's problems might have gone away faster if they had been caused by a simple outright system failure. Then, disaster recovery plans could have brought the company's data back online as quickly as a few minutes later.

Ken Steinhardt, Director of Technology Analysis at EMC, says one EMC customer—a bank in Europe—had a data center failure caused by a bad adapter for a host computer in a storage network. That corrupted the data in its operational databases. Yet, because of a mirrored copy of data created with the company's TimeFinder software, the European bank was able to restore data quickly and had no significant downtime.

This article was originally published on 2002-05-15
David F. Carr David F. Carr is the Technology Editor for Baseline Magazine, a Ziff Davis publication focused on information technology and its management, with an emphasis on measurable, bottom-line results. He wrote two of Baseline's cover stories focused on the role of technology in disaster recovery, one focused on the response to the tsunami in Indonesia and another on the City of New Orleans after Hurricane Katrina.David has been the author or co-author of many Baseline Case Dissections on corporate technology successes and failures (such as the role of Kmart's inept supply chain implementation in its decline versus Wal-Mart or the successful use of technology to create new market opportunities for office furniture maker Herman Miller). He has also written about the FAA's halting attempts to modernize air traffic control, and in 2003 he traveled to Sierra Leone and Liberia to report on the role of technology in United Nations peacekeeping.David joined Baseline prior to the launch of the magazine in 2001 and helped define popular elements of the magazine such as Gotcha!, which offers cautionary tales about technology pitfalls and how to avoid them.
eWeek eWeek

Have the latest technology news and resources emailed to you everyday.