When you have a conversation with security leaders and take notice of what’s generally high on their list of concerns, incident response tends to be near the top. Though, it’s not necessarily the tactical response of today, but instead the continued emphasis to evolve incident response as a discipline. There are countless articles and thought-leadership pieces circling the Internet, but what’s missing is the acknowledgment that we still make decisions based on limited information. I must make the distinction that it’s not because of a lack of data; rather there are no simple ways to make sense of the mountains of security data that exist as a byproduct of the innate need to “collect data.”

For more than 10 years now, there has been a playbook that most organizations have followed as divinely inspired text. The playbook had three principle directives:

1) gather every piece of data you can
2) store that data in a security-specific database
3) give an analyst a username and password

So, we began to monitor everything, received alerts from all over the network, and tried to piece together a coherent tapestry for action.

Nested within each of the recommendations is the assumption that gathering and storing data would be sufficient. I would ask you to consider that this assumption is flawed. If incident response is to evolve as a discipline, we need to shift to a new paradigm. A paradigm that emphasizes knowledge over data.

To be sure, no one is at fault here; many followed the playbook, and it was the right thing to do. Following the playbook’s directives was certainly better than doing nothing. However, incident responders still remain some of the least equipped members of security organizations. Why is that?

For starters, we believed that investments in SIEM and other technologies (directives 1 and 2) would deliver the right data, quickly. We assumed that the primary need was more data in order for responders to make decisions, so we continued to toss more data into the soup. This is flawed.

There’s a need for knowledge-driven decision making. But in order for that to happen, we must analyze the data we have – not just pile more data into the tank. With that in mind, let’s digress for a moment and talk a bit about data, information, and knowledge.


This seems obvious, we need data to know what’s happening. Although data is a good place to begin, it is insufficient for decisions. Data is a single observation which can contain any number of combinatorial fragments. Data is disconnected, not distilled, and is hopelessly fickle about what it tells us. It is like having a single word drawn from a 26-volume encyclopedia – far from definitive.


Information is a compilation of data. Information gets us closer to the goal, but still leaves much to be desired. Information can be misleading and can accentuate a bias we didn’t even know was there. This natural inclination (the confirmation bias) can keep us from seeing the truth of the matter. This giant pile of data we call “information” does not yield the direction we need to make decisions. It is like having pages pulled from the encyclopedia at random and assuming they each tie together in a coherent frame. Unfortunately, this doesn’t happen. It is more like playing Mad-Libs than getting an education.


Finally, we come to knowledge. This elusive pot-of-gold is what we’re striving to get. Knowledge provides purpose and direction. Furthermore, knowledge is strongest when it disconfirms, rather than confirms; knowledge provides a way to rule out false-positives. Now, we have the encyclopedia along with the instructor guiding our understanding.

If organizations wish to develop incident response as a thorough discipline, they must become methodologically knowledge-driven.

The playbook’s third recommendation creates the false assurance that all we need is smart people with access to the ocean of data we’ve assembled. This is where we choke out the opportunity to mature incident response. Ask any incident responder or response team: “Do you have access to data?” Then, “what are you lacking?” The answer: a meaningful way to make sense of all that data.

They have been inundated with alerts, alarms, sirens, and notifications. They do not need any more of them. For knowledge to take the lead, we need intelligent ways to interrogate our data, ask of it questions, stitch together the picture, and take action are required.

In Part 2, I will make recommendations for a new playbook that will center on the methodological ways to create a knowledge-driven incident response discipline, including: triaging alerts, smart indexing, crawling and tagging, removing data that is irrelevant, and how to stop chasing false positives.

If incident response is going to mature, it has to become knowledge-driven.

Josh Mayfield led a webinar on maturing security intelligence. You can catch a replay here: Watch Replay >>