How to make Kibana dashboards using Immediate Insight collected data
FireMon’s Immediate Insight product is a powerful real-time IT data analytics solution. It provides excellent data discovery, search, & analysis capabilities. Furthermore search results can be saved to Immediate Insight Pinboards for dynamic reporting. Immediate Insight is particularly well suited to monitoring and analysis of unstructured data, & unknown issues which have in the past been difficult to find. As a complement to these capabilities, customers may be interested in producing more traditional security and operations dashboards centered around known metrics. Immediate Insight’s architecture (built on top of Elasticsearch) allows use of third party tools, for example open source tool Kibana, for easy dashboarding of both structured and unstructured data metrics made available by Immediate Insight data enrichment.
The following use case paper outlines a procedure to connect Kibana to Immediate Insight, and produce a simple example dashboard (for complete documentation on Kibana please consult https://www.elastic.co/guide/en/kibana/current/index.html ). It is assumed for the purposes of these instructions that Immediate Insight is already installed on another VM, if you have not yet installed Immediate Insight, please contact FireMon for documentation and assistance.
(Please note that Kibana is not a FireMon product, and FireMon LLC provides no official warrantee or support of the Kibana software nor its interoperability with FireMon’s Immediate Insight product)
Step 1 - Prepare Immediate Insight for use with Kibana
By default, the Ubuntu firewall settings on your Immediate Insight virtual appliance block external access to port 9200 which is required by Kibana to connect to the Immediate Insight Elasticsearch instance. The easiest way to overcome this is to edit the following file and change the DROP rule for ports 9200 to 9299, to instead DROP 9201 to 9299. (note: do not edit other access lists rules before consulting with email@example.com) To accomplish this do the following from the Immediate Insight CLI;
sudo nano /etc/network/if-pre-up.d/insight-port-map
(then make the change highlighted in green, then save your file)
(then reboot your virtual appliance to invoke the change)
You will need to know the Immediate Insight search indexes that you wish to Dashboard with in Kibana. The default search index is the ‘main-stream’ repository, however you may have configured others that you wish to use. From the CLI type the command ‘add-user –l’ to see if other repositories exist (make note of them). For example, below we notice the repository gregprivate is available in addition to main.
Step 2 – Install Kibana and modify its configuration file to point to Immediate Insight
You’ll first need to install Kibana (we recommend Kibana 4.1.4 which is compatible with the current version of Immediate Insight’s Elastic Search). Although it may be possible to install Kibana on the same VM instance as Immediate Insight, we strongly recommend that you install Kibana on a separate VM or Server to avoid conflicts. Instructions below are to install Kibana on Ubuntu Linux (we tested with Ubuntu 14.04.2), although Kibana can also be installed on Windows, Mac, or other Linux distributions.
Download Kibana 4 to your Ubuntu home directory with the following command:
Extract Kibana archive with tar:
Open the Kibana configuration file for editing:
Edit the host field, changing it to the IP address where you have Kibana installed, also edit the elasticsearch_url field, changing the localhost part to that IP address of your Immediate Insight (example shown below), then save the file:
Create a /opt/kibana directory with the following command:
Next copy the Kibana files into your newly-created directory:
Kibana can now be started
To start Kibana manually
However it is preferable to start Kibana as a service so that it stay persistent after reboot
Download a Kibana init script with this command (one line):
Now enable the Kibana service, and start it:
Step 3 – Configure Kibana to connect to the Immediate Insight Index
Access the Kibana web interface;
Click the Settings tab. By default it will show the Index name ‘logstash-*’, you need to change this to the Immediate Insight index name – most likely you should start with main-stream (unless you know your data is in another repository). After entering ‘main-stream*’ for the Index name, click the Time-Field name pull down and select ‘rcvTime’ – then click ‘Create’
Under Indices, you will now see main-stream*, select it (on the right you’ll note varies metric fields available from Immediate Insight). Click the green star logo to make main-stream* your default index.
(Optional) – if you have an additional Immediate Insight index (aka repository) that you wish to be able to report on, click the Add New bottom and repeat the process (following along from our earlier example this could be done for gregprivate*)
Step 4 – Configure Kibana Discovery of Immediate Insight data
Click the Discover link, assuming there is data coming into Immediate Insight’s main-stream repository, you should see data events below. Here you also chose Time Filter and Refresh Interval, because Immediate Insight already provides excellent real-time monitoring capabilities (and to reduce unnecessary performance overhead), we suggest starting with a longer Kibana Time Filter such as 24 hours and a Refresh Interval of 15 minutes. (Note: Kibana queries place some overhead on Immediate Insight)
Note: you won’t be able to use Kibana to view data older than the Data TTL available from the Immediate Insight index.
Step 5 – Configure Kibana Visualizations from Immediate Insight data
Although Kibana offers many different ways to create graphs of different formats, the easiest way for Immediate Insight is to find a metric Field, that interests you, click on the Field on the left of the page to expand it, then click the Visualize link below the Field.
You can accept the default graph formats, or edited them as offered. Next Click Apply, then click the Save Visualization Icon, give it an appropriate Title, then click Save.
Repeat the process to create other Visualizations which you may want to put on Dashboards. Note that you can create Visualizations from different Indices (aka repositories), and if you wish these can later be mixed on the same Dashboard, or split across multiple Dashboards.
Step 6 – Create Kibana Dashboards
Your previously saved Visualizations are available to put on Dashboards. Click Dashboards, click the + icon to add Visualizations, select the ones you want, give the Dashboard a name, then click the Save Dashboard icon.
You can now click the Load Saved Dashboard icon to view your Dashboard. Note that you can optionally change the time period of further edit the Dashboard as needed.
Kibana is capable of producing many different styles and variety of Visualization and Dashboards, we have provided but a simple example here to get you started. Full instruction on configuration of Kibana is beyond the scope of this document – but we hope this introduction gives you an idea of what could be possible using the Immediate Insight data.