The SANS study asked respondents what percentage of their incident response processes are automated through the use of purpose-built tools for remediation workflow. Just 16% automate more than 51% of incident response tasks. No wonder attackers go undetected for months or even years. And, no wonder we can’t deliver even the most fundamental answers to what happened in a breach.
Automation tends to spook IT professionals. But you should be more afraid of what happens without it. We discuss automation in depth in our 2014 DevOps Survey report. DevOps is all about automation, and it can be a boon for security. It opens up architectural discussions and forces entrenched IT constituencies into a mature process, getting people to trust in repeatable and reliable automated processes.
Such automation is the only way we can hope to keep up. Ponemon estimates that the process, from breach identification, to knowing the extent of the breach, to remediating the breach, averages one month per incident. If you have a small security team, there’s no way to work on more than one or two incidents. The SANS survey indicates even small incidents take around four hours to investigate — per endpoint. You need automation to survive.
As to incident response, whether the hit was targeted or a drive-by that got lucky, the business has the same questions: How do we know the extent of the damage? What systems were impacted, and what data was stolen? How do we go about fixing this?
“We have no idea” is always the wrong answer. Gathering this information quickly to make an informed decision following an attack is the main goal of your incident response plan (you do have one, right?). One client admitted that after performing a three-month incident investigation with a team of four staffers, plus consultants, she started the meeting with the board of directors and executive management by stating, “We think we got about 50% of this right.”
And we wonder why the executive leadership balks at more funding.
Prevention tools, when they fail, provide no context or visibility into an attack, and without context and visibility, security teams fly into hunter mode, looking for indicators. We’re talking memory dumps, hard drive images, and frantic chants of “REIMAGE IT!” streaming from the security war room every time IT identifies a machine with indicators of a breach. All this hubbub further annoys the business unit leaders, because it kills service availability, and thus employee productivity, as servers are brought down and PC operating systems reinstalled.
A formal incident response plan is essential. We recommend that you look at incident response like disaster recovery — build repeatable processes and test them. Do table-top exercises and real-world scenario role-playing with all the teams that will be involved in an attack, including operations and legal.
Two things to consider: Even with endpoint threat detection products, you may still need to do forensics (especially if you want the data to be admissible in court), and most of the products available on the market are available only for Windows operating systems, so account for that. Also, legal doesn’t know the difference between a Trojan, a downloader, or a rootkit, so create different priorities for different incidents, and build a taxonomy to classify incidents before you’re in panic mode. For example, a low-level incident such as the Zeus banking Trojan may have a response process that is completely automated, whereas dealing with a more advanced threat that accesses a critical asset like your file server may require communication to legal and approval by a manager.
The No. 1 most important process is breach identification. Define specific thresholds to identify when an alert becomes an incident and when an incident becomes a breach. Prioritize understanding what really happened during an attack; this can significantly reduce costs. Imagine having to notify 100% of your customers that you were breached because you don’t have definitive proof that the attacker didn’t access your customer records. With a smart toolset and processes, you may be able to, in almost real-time, know that the attacker had control of two systems but never accessed sensitive records. This information could save millions of dollars.