Computer Security: Your 5-Step Survival Guide

By John Moore  |  Posted 2006-05-15
It's a dangerous world. Every day, thousands of attacks that threaten to corrupt key systems, steal customer data, and otherwise abuse information-technology assets assault U.S. businesses.

The SANS Institute, which provides computer security education and training, estimates that the average Internet network address experiences an attack every 24 minutes. In most cases, it's an unscrupulous hacker trying to infect corporate computers with viruses, worms and Trojans-commonly dubbed "malware."

But Matthew Speare, group vice president and corporate information security officer at M&T Bank, says that in the trenches it's really not any single type of attack that poses the greatest challenge, but rather "keeping up with the sheer volume" of attempted attacks, which "continue to grow at about 25% to 30% a year."

Full-blown incidents reported to the CERT Coordination Center soared to 137,529 in 2003 from just six in 1988. Since 2003, the center, part of Carnegie Mellon University's Software Engineering Institute, has simply stopped counting, reasoning that attacks have become so common that tallying them no longer tells us anything significant about their scope and impact.

Security breaches exact a huge financial toll. Losses of $130.1 million were reported by the 639 respondents to the 2005 Computer Crime and Security Survey, conducted by the Computer Security Institute (CSI), in conjunction with the FBI's Computer Intrusion Squad in San Francisco. Although that figure was down from 2004, losses per respondent increased in two key categories: Unauthorized access to information reached $303,234 in 2005 from $51,545 in 2004, and theft of proprietary information climbed to $355,552 in 2005 from $168,529 in 2004.

As hackers get more sophisticated, they're also highly motivated and more frequently in it for the money than any underground programmers' glory, says Joe Payne, president and chief operating officer at iDefense Security Intelligence Services, a VeriSign company. Today's perpetrators "don't want to be known for attacks and do them under the radar," he says. "A successful attack is one that doesn't get noticed." He cites the Windows Metafile graphics file format exploit, which surfaced late last year and went undetected in the security industry for about three weeks.

Refined threats call for a new security strategy, says Johannes Ullrich, chief research officer at the SANS Institute.

"With a sophisticated attacker, you have to assume they may have ... a way around any particular safeguard you have in place," he says, making it all the more important to have multiple layers of defense, so the hacker must work harder to get around a series of roadblocks.

But software isn't Superman, guarding your systems and laughing as the bullets bounce harmlessly to the floor; in the real world, absent a defined organizational security policy and experienced personnel to carry it out and make intelligent decisions, your company has a big bull's-eye on it, and as the intruders keep firing away, one day they may just hit the mark.

To help chief information officers and chief security officers secure systems in this changing, challenging threat environment, Baseline presents this Security Survival Guide featuring tips and techniques from the nation's top security experts, including information systems experts, corporate security officers and top security consultants. They'll give you a heads-up on how to detect initial attacks, track down the source, enact an incident response plan, deal with corporate management and learn from experience-and, hopefully, help you sleep a little easier when all is said and done.

. 1: Detect the Initial Attack">
Step No. 1: Detect the Initial Attack

  • Set up multiple layers of security
  • Correlate events to connect the dots
  • Keep your eyes and ears open; intuition supplements technology

    The University of Georgia network security system swats away 80,000 to 90,000 would-be attacks every single day. At the Bank of New York, sensors catch millions of security "events" in a month and "we don't even treat the scripts that run out there or worms flowing across the Internet at any point in time as an incident because they are not entering the network," notes Eric Guerrino, the bank's head of information security.

    With all the threats floating around in the cyberjungle, how do you sniff out a serious I.T. security breach? Best defense requires a mix of technology muscle and human interpretive skills. Detection systems are essential tools, but it's up to professionals to make some informed distinctions.

    Guerrino says his bank's incident-response team sizes up threats based on some critical calculations: the probability of imminent attack, the probability that an attack will succeed once attempted and the potential damage of the attack if it proves successful; the location of the potential targets, the host operating systems and their associated vulnerability to the attack; and the sensitivity of the data residing on affected devices.

    What gives an organization the best chance to safeguard itself? The critical elements include multiple levels of traditional and emerging security monitoring tools; an analysis system capable of crunching copious amounts of event data; and the ability to process observations from employees and customers.

    "Security [today requires] a layered approach," says Payne of iDefense Security Intelligence Services. "There is no silver bullet."

    Security Monitoring

    Firewalls and intrusion-detection systems are the old reliables of detection technology. Standing at the intersection of internal networks and the public Internet, firewalls are the established first barrier to external attacks. Intrusion-detection systems, which joined the security force in the late 1990s, monitor networks for suspicious activity. Intrusion-prevention systems go a step further, monitoring traffic and then initiating an automated response, such as dropping a particular packet of data.

    Old-school intrusion-detection systems identify threats based on the signatures of known attacks. But some new threats are too nimble for that: So-called "zero-day" attacks occur at the same time a vulnerability is discovered, leaving no time for the creation and distribution of signatures.

    To address this, security teams have supplemented signature-based systems with behavior-based detection technologies, which establish a baseline of normal network traffic. The systems then search for anomalous patterns-say, traffic coming from a network at a time when no one should be using it-helpful in flagging previously unknown types of attacks.

    Guerrino calls zero-day exploits "our biggest concern." In response, Bank of New York deploys hundreds of intrusion-detection and intrusion-prevention sensors that record events on a daily basis. Its intrusion-detection/prevention systems shield the bank from the vast majority of exploits, and only a fraction of the events warrant a security-breach investigation, Guerrino says.

    The University of Georgia also uses an intrusion-detection/prevention combination, says Stanton Gatewood, the school's chief information security officer. The university operates a Security Operations Center that monitors its intrusion systems around the clock and also minds firewalls, virtual private networks and other security products.

    Correlating Data

    Monitoring systems generate oodles of data on potentially disruptive security events. Intrusion-detection systems, for example, deploy multiple sensors to scan incoming data packets and flag malicious traffic, creating a log of security events. A sensor covering a single network segment may generate 50,000 to 100,000 alerts in an hour, says Sam Curry, vice president of eTrust threat management solutions at CA.

    To find events that are truly cause for alarm, organizations must comb though thousands, if not millions, of snared signals. I.T. security departments "need some way to filter [events] to get them down to a more manageable level," Guerrino says.

    Security experts refer to the parsing of this information for analysis as "correlation."

    Before he or she gets buried under a deluge of security event logs, correlation presents a security analyst with a short list of items to consider. Security information and event management software uses correlation engines to help connect the dots of security events.

    "A single event in and of itself may not mean anything, but aggregating with multiple data points can signify something more sinister," says Speare, the group vice president and corporate information security officer at M&T Bank.

    Speare says automated tools with intelligent correlation are helpful in identifying phishing attacks, for example, in which a perpetrator sends e-mails claiming to represent a legitimate business and directs recipients to a bogus Web site where they are asked to submit Social Security numbers, credit card numbers or other personal information. The e-mails and Web site use authentic-looking business logos to lull the recipients into a false sense of security about divulging information.

    Speare says people who launch phishing attacks tend to be slow and methodical. Someone planning to spoof a banking Web site may visit the legitimate site to pull an image, return several hours later to pull another, and come back once more after that to obtain a third. The fact that images were obtained over an eight-hour period, amid a multitude of other events occurring during that span, may not mean much, Speare says. But filter those events through a correlation engine, and you can put those pilfered images into context-for example, discovering that the pilfered images were downloaded via the same class of Internet Protocol addresses.

    M&T Bank uses NetForensics' Open Security Platform security information and event management product. Tracy Hulver, senior director of product management at NetForensics, says customers can use the company's products to "uncover patterns that native security devices may not uncover."

    "There's a sea of stars out there that have to be connected to find the constellation that matters," says CA's Curry, whose company also markets a security and event information management product.

    Bank of New York first correlates events logged by each detection method-intrusion detection, intrusion prevention, etc.-and then correlates across the different product sets. It's at this second, higher level of correlation that the bank's security group may issue an alert. At this stage, for example, it may recognize that the same kind of exploit is targeting two different Bank of New York locations from the same source network, Guerrino explains.

    But before sounding the alarm, the security analyst monitoring the firewall/intrusion-detection/intrusion-prevention system confers with his or her manager and network operations personnel to discuss the severity of the incident. Remediation may take place here. Only the more serious incidents move up the chain.

    Security groups must be wary of false alarms. New York-Presbyterian Hospital, for example, battled a false positive rate of more than 15% with the intrusion-detection system it had installed, says information security officer Soumitra Sengupta. The hospital has since purchased a behavior-based security appliance from CounterStorm that Sengupta says has minimized the problem.

    Keeping People Alert

    As necessary as technology-based detection systems are, be alert to clues from users and customers, who may give you a hint something's amiss even before your software does.

    Calls to a corporate help desk are as likely to signal the onset of a denial-of-service attack as a security alarm, says Barry Miracle, global security practice leader at BearingPoint, because such attacks can "outstrip the ability of your tools to respond."

    In these attacks, Miracle explains, the time it takes security staff to scrutinize intrusion-detection reports for false alarms works against you. By the time an incident is tagged as genuine, the phones already may be ringing.

    A company's network operations center might also pick up an early distress signal. "Some problems will get reported as a network or a systems issue so it gets to them first," Guerrino added. Callers' complaints about a perceived network performance issue may turn out to be caused by a new Trojan or worm upon examination. That scenario usually plays out for threats never encountered before, Guerrino says.

    A security manager who finds a new threat may consult with an antivirus vendor, security service provider or computer security forum to alert the community and determine the type of attack or malware encountered.

    Early tips that something is awry also come in from business partners. Dave Morrow, chief security and privacy officer for Electronic Data Systems Corp., describes one scenario: Company A gets a call from a business partner, Company B, reporting an attack that appears to originate from someone on Company A's network. Company A investigates and finds that Company B was indeed hacked via Company A's network. But the culprit turns out not to be a Company A employee, but an intruder who broke into Company A's network from the Internet and then attacked Company B. Morrow refers to this process as "looping." Given that "businesses are so much more of an ecosystem," he explains, a partner may well "see something that will be traced back to you."

    Sometimes, the first indication of a security violation may come from a customer you didn't even know you had.

    The University of Georgia's Gatewood recalls a case a few years ago of a student who used the university's computers to sell software on eBay. The school was unaware of the violation of its security and acceptable-use policy until it fielded a complaint call from a buyer.

    "One of the folks who purchased the software got upset ... and called the institution and wanted money back," Gatewood says, noting the buyer objected to the "cheesy CD cover" and general lack of professionalism.

    Dealing with a community of intelligent, creative students who arrive on campus with a small data center's worth of computing equipment poses a challenge for university security, Gatewood says.

    "I've had quiet days," he says, "but 'quiet' is a relative term."

    It's a reminder that insiders, whether student or employee, represent a security threat that may elude intrusion-detection systems, which "typically are deployed to detect events or attempts from the outside coming in," Guerrino says.

    However, intrusion-prevention systems can be deployed to monitor internal activity; the Bank of New York uses them that way.

    M&T's Speare also notes the use of intrusion-prevention systems for internal monitoring, as well as IPS products from Vericept and Vontu that look for specific types of information, including Social Security numbers, traversing corporate networks, he says.

    . 2: Track Down the Source">
    Step No. 2: Track Down the Source

  • Leverage security information and event management software
  • Consider configuration management and automated response tools
  • Scan for vulnerabilities before they're exploited

    When an alert shows up on a security manager's console, it's as if someone set off an alarm, says Morrow, the chief security and privacy officer for Electronic Data Systems Corp.

    The security group's first question is obvious: Where is the problem? But finding the answer requires ingenuity. There's no single surefire method for finding a security breach and nailing down its scope.

    "That task is still more art than science," says Mark Zajicek, technical staff member with the CERT Coordination Center, which studies Internet security vulnerabilities and offers security training. "It's very ad hoc."

    Leverage Event Data

    Event logs generated by firewalls and early warning intrusion-detection/prevention systems give security analysts one route of inquiry. Demand for tools that help correlate the mass of security data held by the various systems is growing, Zajicek says.

    Security experts advise looking at security information and event management software, which helps security managers detect incidents, for clues that may help identify the source of the attack as well.

    Security information and event management software rolls up alerts from firewalls and intrusion-detection/protection systems, along with event data from antivirus products, databases, Web servers and elsewhere. It offers two tracks to get to the source. One is its visualization portion, which looks like a large, continuously scrolling spreadsheet and provides some amount of detail on a network attack, detected virus or other event, including the Internet Protocol address of the affected equipment and device name.

    The initial information gives a basic sketch of the problem and where it may exist. Every device connected to a network is identified by an Internet Protocol address, for example, which can guide security personnel to the general areas requiring investigation, says David Lawson, director of global services, risk management and compliance at Acumen Solutions, a business and technology consulting firm.

    However, there are limitations to this line of inquiry; one is a lack of context. "What does the IP address mean?" asks David Giambruno, director of engineering and security at Pitney Bowes. "Where is it and who is using it?"

    The other limitation, Lawson notes, is that an attack may spoof the IP address.

    Security analysts thus have to dig deeper into the second source, the event logs, which contain more finely grained detail. They'll be looking for Media Access Control addresses, which identify network nodes, to see if a given IP address is correct and valid, Lawson explains. The logs also will provide details on how an attack progressed through a network. By examining the firewalls and routers and operating systems, analysts can piece together how many Media Access Control addresses, Internet Protocol addresses and routers were targeted in a given incident, Lawson says.

    Security personnel, Lawson says, "need information beyond the alert itself." A good security information and event management system will archive logs from different security devices, routers and operating systems.

    To put alerts in context, Pitney Bowes mapped out a "foundation layer" of its information assets. The company keeps tabs on each I.T. device: what it is, what it does, who uses it, and how it is used. The particulars for a single monitored asset break down into a multitude of attributes-900 for each device, according to security director Giambruno-enumerated in a white paper provided by Pitney Bowes, including operating system, applications, services, accounts and users. Overall, the company's I.T. holdings possess 50 million attributes that generate operational and security data to the tune of 120 million correlations each day.

    "I can't handle the scale of [that] data and put it together," Giambruno acknowledges, so Pitney Bowes has outsourced the enormous task to Intuitive Labs, maker of the Operational Excellence security tool, noting that his company is Intuitive Labs' first customer.

    The data "all rolls up into a massive correlation and inference engine," Giambruno says, which pulls in reports from all of Pitney Bowes' instrumentation systems, including security, networks, servers, desktops and directories, configuration management and applications, among others.

    Pitney Bowes receives security status information from Intuitive Labs via a Web portal that displays the company's security situation in graphical form, with devices color-coded by vulnerability. If a critical patch for Oracle databases is announced, say, the company's Oracle server will appear in red.

    "Our entire world is red, yellow, green," Giambruno says.

    The security setup Giambruno describes doesn't come easily. Curry, the vice president of eTrust threat management solutions at CA, says customers' security information and event management deployments generally occur in phases, starting with a pilot project involving perhaps 100 to 200 sources of security events. Those sources may include intrusion-detection/prevention sensors and firewalls. In the next step, an organization expands the system to additional sources of data or parts of the enterprise. The initial pilot, Curry says, typically focuses on networks and systems the customer deems critical.

    Configuration Management

    A security information and event management system's data gives the security team direction; after that, they must still physically find the affected system.

    A configuration management database, which holds information about the components of an organization's information-technology infrastructure, can help. By identifying components and their status, the database helps security managers zero in on the source of trouble, though that doesn't mean all devices are easy to find; a laptop plugged into the corporate network by a temporary worker or other visitor will be elusive.

    For all the automated sleuthing, a certain percentage of devices will be discovered only by "some guy crawling through offices, plugging and unplugging things," Lawson says.

    The security group's charter typically doesn't cover such low-tech snooping. Axel Tillmann, vice president of security vendor Enira Technologies, says most customers he visits assign network engineers to do the poking around, a time-consuming process with critical business implications. He cites a case where an automobile manufacturer detected a security incident but "couldn't find it quick enough." The company had to shut down production long enough to affect the assembly of 300 cars.

    Enira sells "dynamic quarantining" technology, which integrates with security information and event management systems, and automates incident response. Its Network Response Module quarantines affected network nodes. According to Enira, the product determines the location of the device-down to the specific port on a switch-and dynamically reconfigures the appropriate devices to disable the node's network access. The response can also be initiated manually, through a Web browser graphical user interface.

    Chad Dougherty, technical staff member at the CERT Coordination Center, said dynamic quarantining may be more useful in some cases than others. It works when the malicious code has a well-understood impact, he says, but adds that a fully automated response may not be the best approach for dealing with targeted, stealthy attacks, such as the infiltration of a server from which further attacks are launched. "Organizations may not want to deploy [dynamic quarantines] in situations where they absolutely want to be sure what happened," he says.

    As with detecting an attack, human intelligence must support automated systems in determining the scope and severity of an attack. Security managers say they seek out the affected asset's owner.

    "We find out who it is and ask a few questions," says Gatewood, the University of Georgia's chief information security officer. Security personnel will query the systems administrator about the type of applications running on the affected machine and the sensitivity of the data, he adds.

    "We then can make a decision ... to stop there and do a clean or a reload, or take it to the next level," Gatewood says.

    Determining the appropriate response means taking the attack's venom into account. Besides wanting to know how many systems are affected and where, security personnel also seek to determine "the insidiousness of the attack," Lawson says. "Is it a random exploit ... or a botnet propagating through the network and reporting information back to somebody or some organization through an IRC [Internet Relay Chat] channel? Something like that is much more impactful."

    An Internet Relay Chat channel permits the real-time exchange of text messages among users. It also provides the means to control botnets-a group of computers that hackers control remotely and use to transmit spam or viruses-and send captured information back out of a compromised network.

    Assessing Vulnerabilities in Advance

    While corporate security groups chase down incursions when they happen, they've tried to become more proactive, looking for and fixing weak spots before attacks occur with the help of vulnerability management tools. Like intrusion-detection sensors and firewalls, these tools may feed into security information and event management systems and configuration engines.

    Many organizations scan for vulnerabilities on a regular basis, allowing security personnel to determine which systems are vulnerable to attack and patch accordingly.

    At Pitney Bowes, Giambruno says, "we scan everything [for vulnerabilities] all the time" with products from McAfee's Foundstone division (general vulnerability management), Lumeta (networks) and AppDetective (applications). The scanners feed into Intuitive Labs' correlation engine, which flags configurations that could invite trouble or devices in need of a patch.

    In general, vulnerability scanners may also identify trouble spots such as weak passwords and missing patches.

    Eric Hanson, manager of I.T. security at Quad/Graphics, says his company uses Pedestal Software's SecurityExpressions vulnerability and compliance management product to pinpoint areas requiring remediation "to understand our environment and gauge how we are doing." His group also monitors newsgroups and the SANS Institute, among other sources, to keep abreast of news about vulnerabilities.

    Lawson says organizations also run vulnerability scans as an attack unfolds: "As you are starting to see a certain kind of attack, you can plug into a vulnerability management system, see what the potential impact could be and head it off."

    . 3: Enact An Incident Response Plan">
    Step No. 3: Enact An Incident Response Plan

  • Enlist outside intelligence resources
  • Empower a computer incident response team (CIRT)
  • Decide what matters most: The investigation, or halting the attack?

    When a security incident occurs, it's the I.T. security group's job to respond. Among the group's first assignments: Determine whether an alert represents a serious incident or a false alarm. Security managers may call upon internal experts or external help from antivirus vendors and various intelligence services, which provide reports on computer security threats.

    UPS subscribes to a number of such services "that allow us to understand what's going on," says Paul Abels, the company's manager of security policy strategy and business continuity planning. UPS also maintains a strategic relationship with an antivirus vendor. The relationships help UPS stay on top of the threat environment, which puts the company in a position to "react ahead of time," Abels says.

    Reaching Out

    But the knowledge flows in both directions. When UPS discovered a variant of the Zotob worm, the company notified its antivirus vendor. "We were the first ones to report it to them," Abels says. Zotob achieved notoriety in August 2005 when it hit CNN and The New York Times, among others.

    "Typically, [security managers] try to reach out to see if anyone else is experiencing the same thing," notes Miracle, the global security practice leader at BearingPoint.

    An alert that reaches full-blown incident status triggers an organization's response plan-assuming it has one. Security experts say large enterprises typically do maintain some type of formal response plan, though Lawson, director of global services at Acumen Solutions, says incident response varies widely. Some response plans, governed by extensive steps and checklists, become so choreographed that they are "almost restrictive," he says. The other extreme is no choreography, which according to Lawson results in a "mad dance." He suggests a middle path.

    Gatewood, the University of Georgia's chief information security officer, said his institution follows established incident-handling protocols, based on documentation from the National Institute of Standards and Technology (NIST) and the SANS Institute.

    NIST's Computer Security Resource Center publishes a range of security policy guidelines, some of which touch on incident response. The SANS Institute, in conjunction with the Center for Internet Security, offers the Security Consensus Operational Readiness Evaluation, which seeks to provide a minimum standard for information security procedures and checklists. ISO 17799, which provides guidelines for security management, also covers incident management.

    "A number of different frameworks can be used," says Payne, the president and chief operating officer at iDefense Security Intelligence Services. "Good security policy is like religion; it's more important that you have one ... than believe in any particular one."

    Tim Grance, chief of the System and Network Security Group at NIST, says organizations should adapt and modify incident response guidelines to suit their needs. But he added that security groups should document what they are doing so successors will understand the approach.

    Empower a Computer Incident Response Team (CIRT)

    At some organizations, a computer incident response team (CIRT) puts the response plan into action. The corporate security chief generally heads the CIRT, but some companies prefer to tap an experienced outsider to manage response activity, so that one person doesn't wear two hats in a crisis.

    "It's extremely difficult to coordinate all the activities that need to happen out of the security group and lead the CIRT at the same," says BearingPoint's Miracle, who has held security management positions at ADP and Charles Schwab.

    He says outsourcing this responsibility is fairly common at big companies, but unusual at smaller firms.

    The CIRT team consists of I.T. security specialists, either internal or from the outside, and people with other areas of expertise. "The CIRT team has to cross a lot of disciplines," Miracle explains. "The security people can't make changes on the desktops or changes in production systems. CIRT has to drive those changes."

    Miracle says CIRT usually includes desktop gurus, server managers and help-desk representatives. The CIRT members' responsibilities are determined in advance. "In real time, you can't have people arguing ... that you can't shut that server down," Miracle explains. He adds that some companies hire consultants to help establish roles and get different groups across the organization to buy into the plan.

    While the CIRT team may have broad influence, its physical reach may be limited. To address this issue, the University of Georgia's security group has deputized security liaisons in each of the institution's 14 colleges, Gatewood says. Each college has a different security parameter, but through the use of institutional policies, standards and processes, the university has been able to set a security baseline, Gatewood says. A security liaison also represents the university's administrative users.

    Gatewood's security group trains the liaisons to know how to react as an event unfolds. I.T. personnel or business managers may serve as liaisons. The university's triage team, responsible for coordinating incident response, calls the liaisons into action when an incident affects their school or user group.

    Next, the CIRT-or those empowered by the group-takes steps to isolate the affected area and remediate the problem. This could mean anything from shutting a port on a switch to removing viruses from infected workstations.

    Often, an organization will shut down an entire network rather than just shutting a port or throttling bandwidth. For example, Lawson says, a multinational organization experiencing a fast-moving network attack in Brazil or Romania might completely isolate that part of the network while it embarks on remediation.

    For malware cleanup, an organization may choose to reload a fresh software image rather than delete the offending code. Tillmann, vice president of security vendor Enira Technologies, says more companies choose such "brute-force methods" because they find it less arduous than potentially spending hours cleaning infected files from a system.

    "Most corporations have standard computers that allow them to have a default hard drive configuration," Tillmann says. "All they need to do is wipe the hard drive and reinstall the image."

    Consultant Lawson cites the example of a European bank that doesn't even wait for a specific incident to reload images; it periodically reloads the image on every front-facing server, assuming the servers will be hacked at some point.

    Let the Attack Go, or Cut It Off?

    Brute force or otherwise, cleanup comes to a halt when an incident calls for a forensics examination.

    During an ongoing network attack, the organization must decide whether to let the incursion continue to aid its investigation or cut it off to minimize damage. Technology and business leaders must weigh whether "the investigative process outweighs the risk to the network," says Morrow, the chief security and privacy officer for Electronic Data Systems Corp.

    Sometimes it's strictly a business decision. But criminal cases may involve external authorities such as the FBI, Morrow notes, and "they will weigh in with what they want to do."

    Organizations may lack the specialized staff to investigate computer crime. Miracle says forensics is frequently outsourced.

    Bank of New York handles most response tasks internally, but may call in a forensics specialist if an incident "looks like something that might lead to litigation," says Guerrino, the bank's head of information security. An event such as theft of service could spark a forensics investigation, but could also be treated as an employee matter if the theft occurs internally.

    The bank has a retainer-like contract with a forensics services firm that gathers evidence and maintains the chain of custody, Guerrino adds.

    While investigation and remediation activities continue, incident responders, ideally, keep lines of communication open with key constituencies. The CIRT team, for instance, notifies line-of-business managers of a problem so they can inform their customers.

    . 4: Dealing With Corporate Management">
    Step No. 4: Dealing With Corporate Management

  • Limit false alarms, prioritize events and communicate wisely
  • Avoid surprises; pre-negotiate deals and rates with service providers
  • Keep lines of communications with executives open

    Picking up the phone to call the C-level suite ranks as the most delicate part of a security team's communications plan.

    Discernment is crucial in deciding when and how to inform the powers that be. Top executives need to be in the security loop, but the sky will fall on the security officer who issues one too many false alarms.

    Don't Cry Wolf

    "Your false alarm rate to business people has to be low," says Barry Miracle, global security practice leader at BearingPoint. If a security shop warns erroneously more than twice a year, "no one will respond to the next one."

    The experience and intuition of the security manager plays a major role, according to security experts. "A lot of it is judgment," says Morrow, the chief security and privacy officer for Electronic Data Systems Corp., "and knowing what is of interest to senior executives and what's not."

    The University of Georgia's triage team always assesses the scope and severity of an incident before contacting higher-ups, notes Gatewood, the university's chief information security officer. "If it's someone spamming us or sending out spyware, that's not making the 6 o'clock news," he says.

    M&T Bank ranks incident severity on a 1-to-4 scale, with Level 1 deemed the most critical. A Level 1 incident must involve at least one of the following: unauthorized disclosure, modification, destruction or deletion of sensitive information or data; disruption of business continuity and critical business processes or communication; an impact on the long-term public perception of the organization; or identity theft of an individual or group.

    In response to a Level 1 incident, the manager of the resources involved is instructed to cease use of the resources until the bank's incident response coordinator makes contact and provides further instruction.

    At New York Presbyterian Hospital, the priority of an incident rises as a particular segment of a network becomes sluggish, and then escalates up to the point where there is a complete disruption of service, which "has to be reported," says information security officer Sengupta. At the health-care facility, he adds, any incident that could potentially affect patient care must be communicated upward as well.

    "Incidents all get reported," Sengupta sums up broadly, "but not at the level of individual viruses and not every day."

    For Giambruno, director of engineering and security at Pitney Bowes, context counts. An attack involving one application may sound small, he says, but if that application is a key enterprise system that impacts many people, it may become a need-to-know incident.

    Incidents judged not to rate the C-level executives' immediate attention are periodically summarized and presented to them in a group.

    Guerrino, head of information security at the Bank of New York, provides his incident summary to a board-level committee of senior executives every six months. The summary includes the number of incidents by category, including unauthorized access, disclosure, usage, or destruction; loss or theft of information or equipment containing information; service disruptions; and copyright or trademark infringements. Incidents are further classified by impact and severity.

    Don't Ask for Money

    There's one form of communication with executives that security managers try to avoid at all costs: Emergency requests for money to handle a computer security incident. To avoid such requests, some enterprise security groups negotiate terms with a pre-approved list of vendors on standby to help manage an incident, Miracle says. Contracts with pre-approved vendors stipulate rates in advance.

    A company might have an arrangement with a forensics specialty firm, for example. Miracle says these support pacts don't necessarily involve huge sums of money. A forensics firm may dispatch one or two investigators who work for perhaps two to six weeks and produce a written report.

    Bank of New York contracts with a forensics specialist that gives the bank the option to call on the company for a specified number of hours of work per year, Guerrino says. If the bank exceeds the number of hours specified in the contract, there's an agreed-upon rate increase.

    Open Dialogue

    To ensure reasonably smooth communication in a crisis, security groups need to open a channel of communication with management. Having an established foundation for dialogue is crucial to the security officer's effectiveness even in the normal course of business, and more so in an emergency, security experts say. For example, Gatewood reports to the University of Georgia's chief information officer, who sits with the executive team.

    "I'm brought to the table to explain things to them," Gatewood says. "I can e-mail them, call them, and brief them without having to have special permission. A lot of my peers say, 'I don't have that relationship with the executive management team.'"

    Security experts tout a close relationship with the top brass as critical for maintaining a healthy security budget and a corporate culture that values security.

    Miracle notes tremendous turnover among chief information security officers; he cites former security officers he knows who insist they won't take that assignment again.

    But some security officers have established solid executive-level ties. Morrow, who reports to EDS' executive vice president of risk management, feels fortunate to work with an executive team that is very conscious of security issues.

    "They are willing to take my calls and are interested and concerned when there is an issue," he says. "Many of my colleagues find it difficult to get senior executives' ear. They get the shoulder shrug."

    Gatewood lists several reasons why C-level executives may ignore the chief information security officer, including lack of trust in the individual and a perception that security manages are "inhibitors or disablers."

    Regulatory compliance issues have pulled at least some senior executives into the information-technology security camp. Sarbanes-Oxley, which demands documented risk-management processes, has forged "a much closer relationship between the chief financial officer and the security team today than ... there ever was before," says Payne, the president and chief operating officer of iDefense Security Intelligence Services. He says CFOs have familiarized themselves with the security group's processes and systems and have invested considerably in technology to address risk issues related to I.T. security. "[Sarbanes-Oxley] has been good for the security industry," Payne contends. "Controls and processes that make good sense are now mandated by regulations, and those regulations have teeth."

    Security managers, for their part, have been working to build closer links not just to executive management, but to all levels of an organization.

    Hanson, the manager of I.T. security at Quad/Graphics, lists good communications and partnerships within the business as the biggest boons to a successful security strategy. He said his group has liaisons working with the company's technology and software development team and also maintains contacts in key business units and subsidiaries. He says the security group's outreach could be as simple as bouncing ideas for new security policies or technologies off business-unit representatives. The group may also provide assistance in implementing a security system.

    "Most of my team spends most of its time outside of security working with the different organizations," Giambruno notes. For example, they might discuss a particular security development with Pitney Bowes' application development team.

    Collaboration between the application and security groups means that security controls are embedded in software from the beginning, as opposed to being retrofitted after development.

    The corporate legal department and public affairs shop are two other groups beyond the C-level that might be notified about incidents, says Lawson, the director of global services at Acumen Solutions.

    The corporate groups, in turn, will likely have advice for the security team. The University of Georgia maintains a security advisory council with representatives from the human-resources, legal, internal audit and public affairs departments. The university's chief information officer also serves on the council, which offers guidance on security policies and standards, and acts in an advisory capacity during an incident, Gatewood says.

    Tone is important in building cooperation between security and other business units. "It's not, 'The sky is falling,'" Giambruno says. "We are good at getting people's attention in a positive way." Giambruno prefers to share information on security issues and explain to the application development team or the information-technology infrastructure team, for example, how those issues may affect them.

    "Security has this negative connotation that surrounds it," Hanson says, noting corporate security groups at some companies have a "Big Brother" image. He says his group tries to build consensus rather than dictate security directives. "We want the business to see the security team not as a roadblock, but as a security-minded business partner," he says.

    Gatewood says the university environment, in particular, demands communication and consensus building, because "higher education is very slow to change. It's extremely difficult to turn that ship around, if they don't want to be turned around. What I've done here is try to foster and build relationships with students, faculty and staff." Earlier in his career, Gatewood worked with the U.S. Air Force, where security was much easier to enforce. The brass on one's shoulder, he says, determined how many people would listen.

    "The chief security officer needs to be more of a business-minded leader and strategist," Gatewood says. He adds that it is incumbent on security managers to rely on metrics-the percentage of total systems with a current security plan, for example-to make the case for security.

    "It is all about getting to the ... budget table ... early and bring the right skill set, tools, communications and strategy," Gatewood explains. "If the folks who make the budget decisions speak numbers, mission and bottom line, then you must speak numbers, mission and bottom line. You must show them that their business objectives cannot be attained without security."

    But apocalyptic scenarios and tales of past disasters fail to impress business leaders. "Telling war stories doesn't work anymore," Gatewood says.

    . 5: Learn From Experience">
    Step No. 5: Learn From Experience

  • Schedule a post-incident meeting
  • Run vulnerability scans to ensure that systems remain secure
  • Convert lessons learned into training and education programs

    The follow-up to a security incident typically involves a round of vulnerability assessment.

    Security groups check to make sure that the remediation efforts truly eradicated the problem and patched the afflicted systems. Different types of attacks call for different recovery procedures. An unauthorized access incident could involve the attacker gaining root access to a system. If that's the case, the recommended course of action is to change all of the passwords on the system, according to the National Institute of Standards and Technology's Computer Security Incident Handling Guide.

    But organizations "don't always follow all the steps" toward comprehensively recovering and securing a system," says Zajicek, the technical member of the CERT Coordination Center. He works with the center's CIRT development unit, which provides guidance on establishing incident response groups.

    "Changing all users' passwords in a big organization is a very tedious job and a time-consuming and very intensive manual process," Zajicek says. "I don't know how many times someone has said, "I've changed the root password.' That's not sufficient."

    An intruder who gains root access has obtained administrator-level access to the system.

    Run a Vulnerability Scan

    Security teams usually conduct a post-incident scan with vulnerability assessment tools to ensure that necessary actions, such as applying required patches, have been taken. But security managers say they are continuously scanning anyway to uncover vulnerabilities or violations of security policy.

    Hanson, the manager of I.T. security at Quad/Graphics, says he uses Pedestal Software's SecurityExpressions to scan desktops, servers and networking gear for compliance to its security policies. "We can use that information to go back and harden things up," he says.

    SecurityExpressions checks for gaps in several key areas including system security configuration settings, security patches, antivirus status, personal firewall status and industry-known vulnerabilities, according to Pedestal Software.

    Quad/Graphics has customized SecurityExpressions to help assess compliance to the company's acceptable-use policy. The result is an executive-level snapshot in time of whether end users are following policy. The company also brings in an outside analyst every few years to perform a vulnerability assessment.

    The University of Georgia runs vulnerability scans and has vulnerability management applications installed on sensitive and critical servers, says chief information security officer Gatewood. The vulnerability management applications check configurations or settings on servers and generate a report card, which covers areas such as operating systems level and patch, open vulnerable ports and user accounts, Gatewood says.

    "We do vulnerability assessment and scans on a regular basis," adds Abels, UPS' manager of security policy strategy and business continuity planning. The scans, performed by a managed security services provider, may also be scheduled on an on-demand basis as a follow-up to an event.

    A vulnerability assessment is largely a technical exercise. Enterprises also convene post-incident meetings with representatives from different areas of an organization, which focus on process as much as technology.

    Gatewood says his security group holds an "aftermath party" with the university's security advisory council, including the chief information officer and representatives from the legal, public affairs and HR departments, among others.

    The meeting dissects the security team's response to the incident, assessing the effectiveness of processes and procedures, Gatewood explains. The follow-up meeting also serves as a springboard to spread the word about a given incident, with an eye toward avoiding it in the future.

    Learn From Mistakes

    Security experts point to education as the most important safeguard against future incidents. At Electronic Data Systems, an employee undergoes security awareness training when he or she first joins the company and annually after that, says chief security and privacy officer Morrow. Managers are held accountable to make sure all who report to them have gone through the training, which is largely handled online, he adds.

    Morrow says security training crops up in other guises. "We integrate the security messaging and data protection messaging into all of our leadership training," he says. The company also schedules a security awareness week each year.

    The University of Georgia runs a campuswide program, educating students on desktop security and identity theft. To illustrate the latter vulnerability, Gatewood tells the tale of a professor who left his attaché case in the school library with his wallet inside. In less than an hour, Gatewood says, the absent-minded professor's credit card was purchasing airline tickets and other items from an off-campus IP address.

    "People are watching," Gatewood cautions.

    The university's security arm also tutors faculty not to post grades with Social Security numbers on the Internet. In general, Gatewood aims to "educate our staff on how to change business processes from a wide-open environment to something a little more secure."

    The university's training methods stretch from instructor-led training to the occasional security flyer. "There are all sorts of ways," Gatewood says. "It is being an evangelist from end to end."

    Training aims to prevent incidents, but an educated user can also contribute to early detection. "They'll know what not to do ... and they'll know when to call if they see something funny," says Lawson, the director of global services at Acumen Solutions.

    Education initiatives must be flexible, enabling security groups to take lessons learned from security incidents and fold them back into the training regimen. They also study changes in attack types and methods and update the curriculum.

    Bank of New York conducts quarterly threat assessments to close existing vulnerabilities and anticipate new exploits. It reviews its security posture annually with a third party. The bank's new understanding of the threat environment is incorporated into training programs for technical people and awareness programs for the rest, according to Guerrino, its head of information security.

    Keeping information-technology departments up to speed on security is another dimension of the security group's education initiative. Application developers, for example, need to incorporate the organizations' latest security principles as they generate code.

    "When we go down this path to integrate security into the ... development life cycle, we need to have an ongoing training and awareness program for the technical staff," Guerrino points out. Through this program, the bank offers three to five secure-coding classes every year from a third-party training firm.

    Application development teams at Bank of New York go through a security review process to see whether required controls have been implemented, Guerrino says. The code review "makes sure they adhere to controls and documents the reason why some controls may not be applicable to them," he says. The bank uses a third-party firm to conduct the review.

    Ongoing training efforts help keep security on the front burner, say security executives, who warn that the absence of major incidents tends to lead to complacency.

    Companies "that are not successfully attacked get lax and you have to reinvigorate them," says Miracle, the global security practice leader at BearingPoint.

    "Understanding the hazards and risks and threats of doing business in a networked environment," Gatewood sums up, "will help you become much more secure."