Enhancing Metron Dashboard

Now that we have added the Squid Data Source, we want to visualize the data in the Metron Dashboard. The below provides instructions. 

Step 1: Setup and Pre-requisites

1. You should have completed the instructions in Adding a new Telemetry Data Source
2. Make sure the following variables are configured based on your environment: 

   
   KAFKA_HOST = host where a Kafka broker is installed
   ZOOKEEPER_HOST = host where a Zookeeper server is installed
   PROBE_HOST = Host where your sensor, probes are installed. If don't have any sensors installed, pick the host where a storm supervisor is running
   SQUID_HOST = Host where you want to install SQUID. If you don't care, just install on the PROBE_HOST
   NIFI_HOST = The host where you will install NIFI. You want this this to be same host that you installed Squid.
   HOST_WITH_ENRICHMENT_TAG = This is the host in your inventory hosts file that you put under the group "enrichment" 
   SEARCH_HOST = This is the host where you have elastic or solr running. This is the host in your inventory hosts file that you put under the group "search". Pick one of the search hosts
   SEARCH_HOST_PORT = The port of the search host where indexing is configured. (e.g: 9300)
   METRON_UI_HOST = This is the host where your metron ui web application is running. This is the host in your inventory hosts file that you put under the group "web".
   METRON_VERSION = The release of the metron binaries you are working with (e.g: 0.2.0BETA-RC2)

Creating Your Own Dashboard

Now that you understand Metron's default dashboard, let's cover how you might extend this dashboard for your own purposes. We will continue the ongoing example of parsing Squid Proxy logs. The dashboard will be extended to display the Squid log data.

Enhance the Squid Data

The previous tutorials covering Squid produced a limited data set. These consisted of a few basic requests. To make this tutorial more interesting, we are going to need a bit more variety in the sample data.

1. Copy and paste the following set of links to a local file called `links.txt`.

    https://www.amazon.com/Cards-Against-Humanity-LLC-CAHUS/dp/B004S8F7QM/ref=zg_bs_toys-and-games_home_1?pf_rd_p=2140216822&pf_rd_s=center-1&pf_rd_t=2101&pf_rd_i=home&pf_rd_m=ATVPDKIKX0DER&pf_rd_r=2231TS0FE044EZT85PQ4
    https://www.amazon.com/Brain-Game-Cube-Intelligence-Development/dp/B01CRXM1JU/ref=zg_bs_toys-and-games_home_2?pf_rd_p=2140216822&pf_rd_s=center-1&pf_rd_t=2101&pf_rd_i=home&pf_rd_m=ATVPDKIKX0DER&pf_rd_r=MANXEWDTKDH2RD9Y3466
    https://www.amazon.com/Zuru-Balloons-different-colors-Seconds/dp/B00ZPW3U14/ref=zg_bs_toys-and-games_home_3?pf_rd_p=2140216822&pf_rd_s=center-1&pf_rd_t=2101&pf_rd_i=home&pf_rd_m=ATVPDKIKX0DER&pf_rd_r=MANXEWDTKDH2RD9Y3466
    https://www.amazon.com/MAGINOVO-Bluetooth-Headphones-Wireless-Earphones/dp/B01EFKFQL8/ref=zg_bs_electronics_home_1?pf_rd_p=2140225402&pf_rd_s=center-2&pf_rd_t=2101&pf_rd_i=home&pf_rd_m=ATVPDKIKX0DER&pf_rd_r=MANXEWDTKDH2RD9Y3466
    https://www.amazon.com/Amazon-Fire-TV-Stick-Streaming-Media-Player/dp/B00GDQ0RMG/ref=zg_bs_electronics_home_2?pf_rd_p=2140225402&pf_rd_s=center-2&pf_rd_t=2101&pf_rd_i=home&pf_rd_m=ATVPDKIKX0DER&pf_rd_r=MANXEWDTKDH2RD9Y3466
    http://www.walmart.com/ip/All-the-Light-We-Cannot-See/26737727
    http://www.walmart.com/ip/Being-Mortal-Medicine-and-What-Matters-in-the-End/36958209
    http://www.walmart.com/ip/My-Brilliant-Friend-Book-One-Childhood-Adolescence/20527482
    http://www.walmart.com/ip/A-Game-of-Thrones/402949
    http://www.bbc.co.uk/capital/story/20160622-there-are-people-making-millions-from-your-pets-poo
    http://www.bbc.co.uk/earth/story/20160620-can-we-predict-the-time-of-our-death
    http://www.bbc.co.uk/news/uk-england-somerset-36596557

2. Run this command to choose one of the links above at random and make a request for that link through Squid. Leave this command running in a terminal so that a continual feed of data is generated as we work through the remainder of this tutorial.

    while sleep 2; do cat links.txt | shuf -n 1 | xargs -i squidclient -g 4 -v {}; done

3. The previous command is generating log records at `/var/log/squid/access.log`. Run the following command in another terminal to extract this data and publish it to Kafka. Again, leave this command running to generate that continuous feed of data. You will need to have two separate terminal sessions left running.

    tail -F /var/log/squid/access.log | /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list $KAFKA_BROKER_URL --topic squid

4. Ensure that the parser topology for Squid continues to run based on the steps outlined in the previous tutorials.

Create an Index Template

To work with the Squid data in Kibana, we need to ensure that the data is landing in the search index with the correct data types. This can be achieved by defining an index template.

1. Run the following command to create an index template for Squid.

 curl -XPOST $ES_HOST:$ES_PORT/_template/squid_index -d '
 {
 "template": "squid_index*",
 "mappings": {
 "bro_doc": {
 "_timestamp": {
 "enabled": true
 },
 "properties": {
 "timestamp": {
 "type": "date",
 "format": "epoch_millis"
 },
 "source:type": {
 "type": "string",
 "index": "not_analyzed"
 },
 "action": {
 "type": "string",
 "index": "not_analyzed"
 },
 "bytes": {
 "type": "integer"
 },
 "code": {
 "type": "string",
 "index": "not_analyzed"
 },
 "domain_without_subdomains": {
 "type": "string",
 "index": "not_analyzed"
 },
 "full_hostname": {
 "type": "string",
 "index": "not_analyzed"
 },
 "elapsed": {
 "type": "integer"
 },
 "method": {
 "type": "string",
 "index": "not_analyzed"
 },
 "ip_dst_addr": {
 "type": "string",
 "index": "not_analyzed"
 }
 }
 }
 }
 }'

2. By default, Elasticsearch will attempt to analyze all fields of type string. This means that Elasticsearch will tokenize the string and perform additional processing to enable free-form text search. In many cases, and all cases for the Squid data, we want to treat each of the string fields as enumerations. This is why most fields in the index template are `not_analyzed`.

3. An index template will only apply for indices that are created after the template is created. Delete the existing Squid indices so that new ones can be generated with the index template.

    curl -XDELETE node1:9200/squid*

4. Wait for the Squid index to be re-created. This may take a minute or two based on how fast the Squid data is being consumed in your environment.

    curl -XGET node1:9200/squid*

Configure the Squid Index in Kibana

Now that we have a Squid index with all of the right data types, we need to tell Kibana about this index.

1. Login to your Kibana user interface and then click on 'Settings', then 'Indices'.

2. A text field will prompt for the name of the index. Type `squid*` within the text field. Every hour or day, depending on the specific configuration, a new Squid index will be created. Using this pattern will match against all Squid indices for all time periods.

3. Click outside of that text box and wait for the 'Time-field name' input field to populate. Since there is only one timestamp in the index, this should default to a field called `timestamp`. If this does not happen simply choose the field `timestamp`.

4. Then click the 'Create' button.

Review the Squid Data

Now that Kibana is aware of the new Squid index, let's take a look at the data.

1. Click on `Discover` and then choose the newly created `squid*` index pattern.

2. By clicking any of the fields on the left menu, you can see a representation of the variety of data for that specific fields.

3. Clicking on a specific record will show each field available in the data.

Save a Squid Search

Let's create a basic data table so that a user can inspect record-level details for Squid.  In Kibana, this is done by creating a 'Saved Search'

1. Click on `Discover` and then choose the newly created `squid*` index pattern.

2. In the 'Fields' panel on the left, choose which fields to include in the saved search.  Click the 'Add' button next to each field.

3. Click on the 'Save' icon near the top-right to save the search.

Visualize the Squid Data

After using the `Discover` panel to better understand the Squid data, let's create a few visualizations.

1. Click on 'Visualize' in the top level menu.

2. Choose the 'Vertical bar chart' and when prompted to 'Select a search source' choose 'From a new search'. Choose the `squid*` index pattern.

3. Under 'Select bucket types' click the 'X-Axis' and for the 'Aggregation' type choose 'Terms'.

4. Under 'Field' choose the `domain_without_subdomains` field.

5. Click the 'Play' button to refresh the visualization.

6. Near the top-right side of the screen click on the 'Save' icon to save the visualization. Name it something appropriate. This will allow us to use the visualization in a dashboard later.

Customize the Dashboard

1. Open the Metron Dashboard by clicking on 'Dashboard' in the top-level menu.

2. On the right, click the 'Add' button indicated by a plus sign.

3. Find the visualization that you would like to add.

4. Scroll to the bottom of the dashboard to find the visualization that was added. From here you can resize and move the visualization as needed.

5. Continue enhancing the dashboard by adding the 'Saved Search' that was previously created.

Summary

At this point you should be comfortable customizing a dashboard as you add new sources of telemetry to Metron. This article introduced Metron's default dashboard that is built upon Kibana 4. It covered the elements present in the dashboard and how you can extend the dashboard for your own purposes.