Trial by FireBy Samuel Greengard | Posted 2012-06-04 Email Print
Re-Thinking HR: What Every CIO Needs to Know About Tomorrow's Workforce
Disasters and business disruptions are inevitable. An effective disaster recovery and business continuity strategy encompasses people, technology and processes.
Trial by Fire
Protecting data and ensuring that systems operate during a disaster is critical for Quarles & Brady, a law firm with more than 1,000 employees and nine offices in four states, as well as the District of Columbia and Shanghai, China. Two of its offices are located in Florida and have been subjected to hurricanes. Overall, the law firm has approximately 350 terabytes of data and the volume is growing by about 35 percent annually, according to Rich Raether, manager of network engineering.
In the past, if a problem occurred, Quarles & Brady relied on attorneys and staff located in offices to make backups of key files and send them via FedEx to its data centers. "It was not effective and it wasn't robust," Raether says. But with close to 500 applications in use, protecting data and avoiding downtime is paramount.
As a result, the company adopted a platform based on Dell EqualLogic SANs to store data from nearly 400 virtualized Windows-based servers (95 physical servers) scattered across the company. It now replicates data between two data centers in Milwaukee and Phoenix. In addition, it has a redundant Multiprotocol Label Switching (MPLS) network in place.
The environment has provided enormous benefits. When the IT department recently tested the system, it completed the fail-over process in about 45 minutes. And when a tropical storm hit the Naples, Fla., office a few years ago, it took about an hour and 15 minutes to get the office up and running again.
No less important: Quarles & Brady is able to push its DR plan and processes out to individual offices automatically. The result has been a 93 percent improvement in local recovery point objectives and 12-fold improvement in the speed of recovery for lost files. The firm also has achieved a 10-fold improvement in time to provision virtual desktops.
Testing systems and ensuring that they can meet real world needs is at the center of an effective DR and BC strategy, Deloitte’s Sarabacha notes. "Vendors’ claims about resiliency and theoretical numbers about recovery time mean very little,” he says. It is critical to validate a system within the context of realistic conditions. He adds that it's wise to stress and test systems using "war gaming" exercises that throw different variables into the picture. "Only then is it possible to understand how to enhance systems and build the best possible solution," he concludes.
Building on Success
The nature of business continuity is changing. For years, organizations mostly performed operational backups on a seven-to 14-day rotation, with monthly, quarterly and annual data slotted into an archive. Then SANs arrived on the scene and made it easier to store, replicate and restore data across a network. Over the last few years, these capabilities have grown and SANs have become far more sophisticated. Now a growing number of companies are turning to the cloud to address DR and BC challenges.
Graniterock, a 112-year-old construction and construction materials company headquartered in Watsonville, Calif., is among the organizations embracing the cloud. It operates 22 locations, including a chain of retail stores, in Northern California.
Altogether, the firm has five physical servers and approximately 150 virtual servers. These systems access data centers in Seattle and Denver through MPLS and AT&T networks. The company runs its enterprise resource planning (ERP) system and other applications in the cloud. "This greatly reduces the risk during an earthquake, fire or other disaster," notes CFO and CIO Steve Snodgrass.
The company turned to Velocity Technology Solutions to create a cloud-based framework for DR and BC. It also relies on SAN storage devices to store data internally and to provide redundant e-mail and file backups. "If systems go offline, the user community will have no clue that a failure has taken place,” Snodgrass says, because “the system takes constant snapshots of the data, stores it offsite and creates redundancies."
Graniterock prioritizes which systems come up first after a disruption and uses tape backups as the system of last resort. "They are there only in the case of an extreme failure," he explains.