Disaster Recovery Plan Guards City Infrastructure

 
 
Posted 2013-11-19
 
 
 
disaster recovery

By Johnna Shamblin

The city of Mount Dora is located in central Florida along Lake Dora. It has about 12,000 residents and nine utility departments that require a strong IT support system: police, fire, water and wastewater, city manager and clerk, public works, library, utilities, planning and development, and personnel.

Mount Dora's IT department supports the entire employee base and is mostly virtual, with only three physical HP servers. Our critical applications include SunGard (financial application), Microsoft Exchange, USA Software (police-incident management) and a fire-incident management application.

As IT manager for Mount Dora, it’s my job to protect against downtime that could devastate our critical infrastructure and shut down the business functions for one or more of the city's departments.

Although Mount Dora is largely unaffected by coastal Florida's destructive hurricanes, we are not impervious to hardware failures, human error and other more-common contributors to system crashes. Although we have not yet had to face a critical situation, having a reliable disaster recovery system in place is vital to prevent potential IT shutdowns.

After four years of using our disaster recovery system, we found that it wasn't going to be a reliable long-term solution because of its storage limitations. As our appliances began approaching capacity, they were unable to back up large snapshots of our servers. If we lost those servers, recovery time could take a day or longer—as long as it would take to transfer the data to a new server.

Additionally, testing the system was a time-intensive process, taking the team several hours to test just one server. As a result, we were able to run these tests only once or twice a year. With time running out and a short-term solution in place to maintain the servers until a new system could be deployed, we began our search for a new disaster recovery solution.

We had contacted IT counterparts in other cities to find out what they were using. Quorum's name came up frequently, so we decided to investigate them, as well as other vendors. Quorum stood out for us because it was easy to manage and offered automated and on-demand testing capabilities, as well as a one-click solution to boot up recovery nodes when needed. The competing solutions we looked at didn't offer backup testing or a physical appliance.

We knew the problems we faced and wanted a simplified solution to help manage a complex process. So we adopted the Quorum solution at the end of 2012 and deployed it in early 2013. Deployment and installation were easy and straightforward, and we now have the system protecting 25 of our servers.

The system runs an automatic test on a daily basis, and, after testing the recovery nodes, the system provides an automatic report by email. As an extra precaution, we also run manual tests of the recovery nodes and restore individual user data about once a month. Through these tests, we know the system is working perfectly.

We also found that we could configure the appliance to meet the needs of our environment. While running the installation and configuration process stage with a Quorum IT representative over the phone, we decided to split the appliance into two partitions. Because we run a substantial number of small application-type servers, the appliance can double the work done at the same time.

Overall, the city’s experience in using Quorum has been simple and successful. We feel that our infrastructure is secure and can be restored within minutes if an IT issue were to occur.

Johnna Shamblin has been the IT manager for the city of Mount Dora, Fla., for 14 years. She was previously employed by Weisco Computers and the city of Eustis, Fla. She is a Certified Chief Information Officer (CCIO) and a mentor to students in the Certified Chief information Systems program at Florida State University.