Centralizing Windows Logs in JSON with Security Analytics
In this post I will show how you can centralize your enterprise-wide Windows logs with zero cost and via one agent to Immediate Insight - security analytics for data discovery. We will output the logs in JSON (they show up a lot more rich than any other method that I have used - a huge improvement over Snare and others). You will use Microsoft's Windows Event Collector to centralize all of your Windows logs in your domain to one server and NXLog Community Edition to send those logs in JSON format over TCP syslog to Immediate Insight. Immediate Insight's Elasticsearch backend and built-in analytics make hunting, incident response, and general troubleshooting and triage extremely fast, easy, and straightforward.
Setup the Log Sending Hosts
On each log sending Windows server:
Enable the Windows Remote Management Service on the Windows servers that will be sending logs to your central Windows server.
Get to a command prompt and type winrm quickconfig - in this example my Windows Server 2012 R2 Standard x64 server was already setup.
Head to the Local Users and Groups and edit the Event Log Readers group to include the server that will be running Windows Event Collector to centralize the Windows logs (you will need to edit the Object Types to include Computers in order to find the computer successfully).
Head to the Windows Firewall configuration section in Settings and go to Allowed apps - edit as shown to allow only domain-based access to your events.
Setup Your Central Windows Log Repository
On the server that will run Windows Event Collector:
Enable the Windows Event Collector service on the Windows server that you are going to use to centralize all of your Windows logs.
Get to a command prompt and type wecutil qc and follow up with a y for yes to finish configuring the service.
Open Event Viewer on the Windows Event Collector server - go to Subscriptions and select Create Subscription...
Now - edit the Subscription Properties:
Give your subscription a name (mine is Remote Windows Event Log Collection. Set the destination log to Forwarded Events. Hit the radio button for collector initiated - then select the computers/servers that it is to collect logs from (alternatively, you can also use Group Policy to set it up so each source Windows log server sends logs).
Under events to collect - hit the Select Events... button. My settings are shown. If you have a lot of servers, you may want to setup multiple collectors (or perhaps use the aforementioned source computer log send via GPO).
The system warns you that you're being a bit ambitious. :)
Hit the Advanced... button and verify that the User Account radio button is set to Machine Account. Set the Event Delivery Optimization to Minimize Latency. You should be done configuring your subscription/collector.
Check the status of your subscription as shown:
Pop into the Forwarded Events view and you should see logs populating in from your servers.
Setup the JSON TCP Syslog Sender
Now, install NXLog Community Edtion - the default setup is fine. You will edit the nxlog.conf file located in C:Program Files (x86)
xlog to customize how your logs are sent. I have two tested and working conf files available for download here (you will want to edit the settings to match your environment). One is for logging in JSON format to disk and one is for sending in JSON format over TCP syslog to Immediate Insight (which is what I'm using for this blog - and it is shown below).
Restart the service once you have successfully edited your conf file - check the NXLog log (heh) at C:Program Files (x86)
xlog.log - you should see a message like below and start to see logs in your Immediate Insight.
The logs come in nicely via JSON in Immediate Insight:
Now for the fun stuff - using Immediate Insight's built-in security analytics (to bubble up interesting information in your data) and extremely fast search to explore your data with ease.
Tie it Together with Analytics
This looks at frequency in your data - what is happening frequently and infrequently in several categories - along with doing automatic geolocation on all IP addresses upon data ingestion. Find outliers and explore them rapidly.
This takes all of your data for the given time period that you're analyzing and places it in buckets based on the commonality within said data. A way for humans like yourself to use your business intelligence to analyze a large data set very quickly.
Activity and Change
You can also take a look at your data sets and compare them to other periods of time - how do they differ from the last hour, yesterday, etc. and what's new, missing, unchanged, and what is trending up and down...highlighting the anomalous very quickly.
Guilt by association - what other data points has the data that you're looking at interacted with in the period that you're investigating?
Next: Sysmon Integration for Sexier Data
Thanks for reading. In my next post - I will take a look at how we can use Windows Sysinternals' Sysmon to make your Windows logs far sexier - logging things like process creation, hashes (SHA1, MD5, SHA256, or IMPHASH), loading of drivers and DLLs, disk/volume raw access, network connections, change in file creation time, and more! Create a force multiplier by using that information along with Immediate Insight's security analytics...