City of San Diego Upgrades Security, Slashes Risks

Big cities are juicy targets for hackers, and the city of San Diego is a prime example. Ranking as the eighth-largest U.S. city, with nearly 1.4 million residents and a $4 billion annual budget, the city experiences up to 1 million cyber-attacks a day.

Keeping digital attackers at bay is a monumental task that falls to Gary Hayslip, deputy director and chief information security officer for San Diego’s department of IT. Prior to his arrival three years ago, a city-owned nonprofit was providing IT service, and no one was handling enterprise-wide security or risk management.

Not surprisingly, there were glaring gaps in the lines of defense. As many as 250 computers a month were being infected. “Every time that happens, a user is down for a day to a day and a half,” Hayslip says, “and you lose data as well as time.” Lost productivity costs about $600 per incident.

There was a lot of data at risk in the city’s 24 discrete networks, which span 40 departments—everything from police departments to transportation, public safety and more. In addition, San Diego has connections to many outside networks.

“We have connections to the FBI, Interpol and outside vendors, plus many other organizations,” Hayslip says. “Hackers might not be targeting us, but looking for a way into the entities connected to us.”

A Security Solution That Grows With the City

In 2013, the city moved to a managed service approach, contracting out IT functions, and hired Hayslip to head up security. His first order of business was an in-depth audit that took six months. The assessment showed that a lack of visibility was a key issue.

“We have over 40,000 end points connected to our networks,” he explains. “Every department was updating and adding new technologies, so we didn’t have a stable risk line. We needed continuous monitoring, scanning and remediation with 24/7 visibility.”

Hayslip proceeded to build a security operations center and started looking for a solution that could grow with the city. “The biggest thing I was looking for was a flexible, resilient platform built to run on large and small networks that were undergoing heavy change,” he reports.

The ability to integrate with other systems was another must. The city’s IT environment combines old and new technologies, including legacy applications and intelligent smart city devices. “We had to make sure they could all see and talk to each other,” he says.

About 18 months ago, San Diego started working with Tenable Network Security, a company Hayslip had successfully worked with during his years at the Department of Defense. He ran a proof of concept for Tenable’s Nessus system for several weeks so his staff of four could become familiar with the dashboards. The trial was a success, and implementation began, with a mix of in-house and virtual servers as well as cloud services.

The city saw results almost immediately. “We found machines that needed patching, updates or reconfiguring to prevent infection,” Hayslip recalls. “We started redesigning certain portions of the network for better resilience. We went from 250 infected machines a month to 35, which saves us about $1.3 million a year.”

It took Hayslip’s team a while to fully understand the impact of the new tools. “We found that some scans and tests could be intrusive and knock things offline,” he says. “For example, if a water treatment facility sees that it is being scanned, it could shut down. We had to fine-tune the system to our environment, which includes many internet of things applications.”

Integrating With Other Systems Through APIs

San Diego has integrated the Nessus platform with other systems through APIs. “It works well with any security suite,” Hayslip says. The city is currently expanding the system to address PCI (payment card industry) compliance and is laying plans to build a hybrid cloud environment. He is impressed with the system’s flexibility, noting, “We can configure scans to look at different parts of the network, slice and dice, get very specific and generate different reports.”

The constant, in-depth visibility into networks and data flow is a huge plus. “When you have an emergency, you know how it’s all put together, where the important data is,” Hayslip says. “I told the mayor that we’ll never be 100 percent secure, but we want to be able to absorb an attack and still provide services to our citizens.”

With no end to cyber-attacks in sight, San Diego will be relying on Nessus for some time to come. “We just renewed our contract with Tenable for one year, and I plan to negotiate for a three-year contract,” Hayslip says. “I want to keep it.”