So much of the current industry buzz is about the promise of machine learning and artificial intelligence to finally win the security battle. From what I’ve seen, the near term promise of machine learning is to scale the human layer of data analysis for security. Humans are ultimately much better at free association than any rule or algorithm. Applying machine learning and other analysis techniques to human commentary about the data can be a real win for security teams combating a more sophisticated adversary operating in a more complex infrastructure.
I believe security teams need an integrated “social” framework to capture the human assessment. A framework that enables operators to inject context directly to, and collaborate in, the machine and human data used for threat detection and incident response. In social media (Facebook, Twitter, etc.) users can contribute personal commentary and associations by simply adding a tag or note. However, our security teams and systems are in the dark ages with regard to flexibly adding commentary and institutional knowledge directly to the data.
Today’s systems are built for the known, repeatable and structured. They lack the “social framework” necessary for teams to productively engage the unknown and unstructured. For these systems, the primary information-sharing medium for unusual and suspicious activity is messaging applications like email that disconnects assessment and commentary from the data. In fact, one CISO recently shared she is frustrated that institutional knowledge and context for anomalous activity is difficult to leverage across the organization because it is contained in many team member’s PST files.
If security operators and analysts are able to add their personal observations directly to the data that others can leverage, it creates a more collaborative problem-solving environment required for more elusive security threats. Enabling analysts and operators to tag interesting records or data points, the system can then capture that context and leverage analytics to improve results returned by the system.
For tagging to be useful, it must be simple and flexible. The user shouldn’t have to operate in some rigid structure, typical of legacy systems, that limits flexibility and reduces its value. In practice, adding tags and notes needs to be as easy as Facebook and Twitter – a fluid part of user’s normal workflow to:
- Add commentary and assessment
- Create custom data segmentation
- Classify unstructured data
- Correlate structured and unstructured data
The associative and accumulative value of tags increases with use. For example, a user searches on a tag or comment, creating a correlated and working dataset, defined by the tag. This enables teams to see a correlated view (across structured and unstructured data) of non-obvious threats and operational issues.
- Common addresses, users, etc.
- Unusual locations
- Common and uncommon events
- Comparisons across arbitrary time periods
- Reputations and cohorts
Tagging is an important part of Immediate Insight’s new natural language, associative and accumulative approach to security analytics. The system is not built to structure and store the data as fields with direct details, as traditional data systems have done, instead it’s built to track entities, their associations and store information about their reputation (what they’ve done in the past and what other things they are associated with). This associative and accumulative approach enables the system to learn as it sees new data and enables queries that return much better insight.