Threat hunting remains an undeveloped competency for far too many organizations. When surveyed, security professionals confess an overall lack of competency to detect and respond to advanced attacks that slip through their defenses.
In my experience, many organizations still rely on alerts from a SIEM (among other prevention systems). Most security teams will painstakingly build models for indicators of compromise, receive alerts from their SIEM, and “do the best they can” to eliminate the intrusion. What are the results?
- Dwell times continue to remain above 200 days
- Number of incidents detected remains flat (it should be going up)
- Time to respond has increased
- Security analysts have been transformed into alert analysts
What follows is my best attempt to awaken a profession. Awaken a profession to the reality that off-model threats should be our greatest concern. If you want to hunt, you must dispense with conforming everything to the SIEM – it just doesn’t work.
The problem with SIEM deployments is that they do not reach the promise we were sold. Most organizations spend millions of dollars, thousands of man-hours, and small fortunes on professional services only to attain no meaningful advances in hunting.
Why is that? Let’s take a look at the features of SIEM that have left most organizations just as feeble as before.
- IT/Security moves fast, SIEM moves slow.
The original goal was to send every bit of data to the SIEM. But getting that data into the SIEM is no easy task. Each network has myriad types of information it can send to a SIEM, but the SIEM must compress that information into its established structure. Anyone who has spent an afternoon working with data understands that data is less cooperative than a starved mongoose. All networks have data coming from operating systems, devices, firewalls, and machines that do not conform to the SIEM requirements.Parsing becomes the daily work of hunters, response teams, and data scientists. This translation task keeps us from doing the actual work of mining data and hunting threats. It is no wonder that we continue to see dwell times over 200 days. We parse data while attackers compromise our networks. This is madness.
- SIEMs are dumb.
There are two inherent limits to a SIEM, 1) The data compression necessarily leaves out pieces of data that don’t conform to the SIEM’s structure and 2) Closed query is the only way to ask the SIEM questions.Literally, we open and close parenthetical statements in a query language. So, now that we’ve spent our day parsing so our data can go into the SIEM, we then restrict our searches to things the SIEM understands. Asking a SIEM to show you something is very difficult – especially in a hunting context where you want to cast a broad net. Imagine casting a fishing line into a spot on the water and getting nothing. Would you conclude there are no fish in the ocean? That’s what we do with SIEM queries. Of course, you can apply rules, “show me each time a user connects to a known bad site,” but do you realize what has to go into the SIEM to make that possible? Hours of data compression, trial and error to build the right query, and you have to know the variables to begin with – which is not hunting. If the SIEM hasn’t established the variables you are asking, it can’t show the results. It simply does not know.
Correlating all the disparate data across multiple platforms (DLP, IDS, AV, Firewalls, Netflow, etc.) does not come easy to a SIEM. A SIEM doesn’t speak those languages, a SIEM knows its own language. Drawing meaningful, previously undetected patterns is beyond the reach of a SIEM.
- SIEMs don’t scale with the threat horizon.
Related to the previous two concerns, SIEMs don’t scale at the same speed as the adversary (internal or external). By the time the data is collected, parsed, normalized, modeled, and queried, the game has changed. Hunters rely on their human reasoning faculties to detect the nuances and subtleties the SIEM is designed to avoid – SIEMs try to model the known; hunters seek the unknown, off-model threats.Hunters have become butlers to the SIEM, responding to its demands with little freedom to care about what’s actually happening. None of the SIEM vendors have found a way to analyze in real-time to keep up with the evolving tactics, techniques and procedures (TTPs) of adversaries. Discovering new TTPs and seeking places with advanced TTPs are occurring is the hallmark of hunting – SIEMs cannot keep pace.
- SIEMs don’t report what matters.
Ask anyone in the threat hunting discipline, “Does you SIEM give you the information required to make confident decisions?” At this point, you should know the answer. Take endpoints as an example, SIEMs don’t receive information from the endpoint in a meaningful way or scale with the volume of logs from those endpoints. Secondly, SIEMs do not make good use of identity information, where 60% of breaches occur. If you’re primary hunting vehicle is the SIEM, you are hunting without a compass or a map.
- SIEMs cost too much.
Though this is widely known, the ancillary costs of SIEM are stupefying. First, we have capital outlays in the millions, followed by man-hours to ingest data, then we have the overage bills that looks eerily similar to mobile carriers from 10 years ago. You cannot hunt effectively when you are tormented by the costs associated with putting new content into the SIEM. Plus, once you do put new data into the SIEM, you are will not extract the outputs you seek (see above). It’s the definition of waste.
SIEM vendors have not provided the market with an analytics capability, and given their fundamental design, we will not see threat hunting flourish using SIEM. In Part 2, I’ll make the case that for good threat hunting, the best way forward is working with Big Data principles, open-ended search, parsing-free data ingest, and methods for taking the fight to the adversary.