Leveraging Big Data and Cloud for Better Security

As the sophistication of information security breaches continues to grow, organizations realize that they need to do a better job of identifying threats and preventing them from causing data loss and other damage. These companies are exploring a variety of technologies and techniques to bolster their security, including two emerging areas: big data analytics and cloud-based security services.

“Information security needs radical rethinking,” says Steve Wilson, vice president and principal analyst at Constellation Research. “The lessons of data breaches over the past few years are stark Inc. Some of the companies affected by advanced persistent threats and by new hardware attacks were probably doing the best they could.”

Status quo approaches to security are not working anymore, so information security executives need to look for alternative solutions. Many will decide to work with managed security service providers and cloud-based security services, according Jon Oltsik, senior principal analyst at research firm Enterprise Strategy Group (ESG). “Others will bolster their security infrastructure with new controls on end points and networks, as well as better security analytics.”

Clearly, big data—the enormous stores of information that companies are gathering from a variety of structured and unstructured sources—holds much promise as a security tool. And more companies are considering security data as part of their big data efforts.

According to a 2013 report from ESG, 44 percent of organizations surveyed said security data collection and analysis would be considered big data within their organizations, while another 44 percent said they would likely consider security data collection and analysis as part of big data within the next 24 months.

There is a growing volume of security data, ESG says. In the early 2000s, security data collection and analysis focused on network perimeter devices, such as firewalls and intrusion detection and prevention systems. Over time, security analysts expanded data collection to include internal network devices, servers, applications and databases, the firm says.

Newer IT initiatives such as mobile technology, cloud computing and virtualization have added to security data collection. As a result, 86 percent of organizations collect either substantially more or somewhat more security data today than they did two years ago, according to the ESG research.

Using Big Data Analytics for Security

At Automatic Data Processing (ADP), a provider of business processing services for payroll, human resource and other areas, big data analytics for security “represents significant opportunities that we are manically focused on,” says VJay LaRosa, senior director of converged security architecture.

Two key factors have driven the company to use big data for security: the capabilities of adversaries have evolved significantly, and the surge of interconnected devices and the increased reliance on the cloud have created a significant increase in security-related data.

“We are accumulating events at about six billion logs per day in our warehouse, front-ended by a complex event-processing engine,” LaRosa says. “We are actively consuming this data in a relational fashion in a massively parallel data warehouse in order to support the converged nature of our threat management and monitoring programs.”

The data ADP is accumulating comes from security tools, networks and business transactions. As part of its big data security effort, the company is building an 80-node Hadoop cluster for unstructured data storage, with an integrated relational database sitting on top of the Hadoop cluster.

“We are working on developing new real-time streaming analytics with in-memory profiling, coupled with a new Complex Event Processing Engine,” LaRosa says. “We are also working to leverage the built-in, open-source machine learning capabilities that exist in these big data platforms to help advance our capabilities and protect our clients’ funds and data.”

ADP doesn’t disclose the specific products or vendors it’s using, but LaRosa says the company has been using its first-generation platform for about two years. “We are actively building the second-generation infrastructure, which will enable expanded capabilities,” he says. “This new infrastructure will allow us to grow and scale at the size and speeds we need in order to keep pace with this rapidly changing environment.”

The big data analytics efforts have allowed ADP to collect and store raw logs at massive speeds, parse and query the collected logs at speeds supporting critical investigations, and embed analytics into the stored logs to detect malicious patterns or abnormal behaviors.