Page History

Now that we have added the Squid Data Source, we want to visualize the Metron configured to parse, index and persist telemetry events and Nifi pushing data to Metron, lets now visualize this streaming telemetry data in the Metron Dashboard. The below provides instructions. UI. We will be adding 3 new panels to visualize the Squid Events: Histogram Panel, Count Panel and Detail Panel

Step 1: Setup and

...

Prerequisites

1. Complete You should have completed the instructions in Adding a new Telemetry Data Source
2. Make sure the following variables are configured based on your environment: 

    

   • KAFKA_HOST = The host where a Kafka broker is installed.
   • ZOOKEEPER_HOST = The host where a Zookeeper server is installed.
   • PROBE_HOST =
   Host
   • The host where your sensor, probes are installed. If don't have any sensors installed, pick the host where a
   storm
   • Storm supervisor is running.
   • SQUID_HOST =
   Host
   • The host where you want to install SQUID. If you don't care, just install SQUID on the PROBE_HOST.
   • NIFI_HOST =
   The host
   • Host where you will install NIFI. You want this this to be same host
   that
   • on which you installed Squid.
   • HOST_WITH_ENRICHMENT_TAG =
   This is the
   • The host in your inventory hosts file that you put under the group "enrichment." 
   • SEARCH_
   HOST
   • HOST =
   This is the
   • The host where you have
   elastic
   • Elastic or
   solr
   • Solr running.
   This
   •  This is the host in your inventory hosts file that you put under the group "search". Pick one of the search hosts.
   • SEARCH_HOST_PORT  = The port of the search host where indexing is configured. (e.g
   :
   • ., 9300)
   • METRON_UI_HOST =
   This is the
   • The host where your
   metron ui
   • Metron UI web application is running.
   This
   •  This is the host in your inventory hosts file that you put under the group "web."
   .
   • METRON_VERSION = The release of the
   metron
   • Metron binaries you are working with. (e.g
   :
   • ., 0.2.0BETA-RC2)

Step 2: Create More Squid Sensor Data

...

1. Run the following command to create an index template for Squid. 

    curl -XPOST $SEARCH_HOST:$SEARCH_PORT/_template/squid_index -d '
    {
    "template": "squid_index*",
    "mappings": {
    "brosquid_doc": {
    "_timestamp": {
    "enabled": true
    },
    "properties": {
    "timestamp": {
    "type": "date",
    "format": "epoch_millis"
    },
    "source:type": {
    "type": "string",
    "index": "not_analyzed"
    },
    "action": {
    "type": "string",
    "index": "not_analyzed"
    },
    "bytes": {
    "type": "integer"
    },
    "code": {
    "type": "string",
    "index": "not_analyzed"
    },
    "domain_without_subdomains": {
    "type": "string",
    "index": "not_analyzed"
    },
    "full_hostname": {
    "type": "string",
    "index": "not_analyzed"
    },
    "elapsed": {
    "type": "integer"
    },
    "method": {
    "type": "string",
    "index": "not_analyzed"
    },
    "ip_dst_addr": {
    "type": "string",
    "index": "not_analyzed"
    }
    }
    }
    }
    }'

2. By default, Elasticsearch will attempt to analyze all fields of type string. This means that Elasticsearch will tokenize the string and perform additional processing to enable free-form text search. In many cases, and all cases for the Squid data, we want to treat each of the string fields as enumerations. This is why most fields in the index template are `not_analyzed`.
3. An index template will only apply for indices that are created after the template is created. Delete the existing Squid indices so that new ones can be generated with the index template. 

   curl -XDELETE $SEARCH_HOST:9200/squid*

4. Wait for the Squid index to be re-created. This may take a minute or two based on how fast the Squid data is being consumed in your environment. 

   curl -XGET node1:9200/squid*

...

1. Login to your Kibana user interface and http://METRON_UI_HOST:5000 and then click on 'Settings', then 'Indices'.

...

3. Clicking on a specific record will show each field available in the data.

Step 6:

...

Adding Squid Event Count Panel to Dashboard

1. Log into the Metron UI Dashboard: http://METRON_UI_HOST:5000
2. Select "Visualize" Tab --> Select "Metric" Visualization"= --> Select "From a new search" for Search Source --> Select "squid*" index source –> Click the Save disk icon on the top right
3. Name the Visualization "Squid Event Count" and click Save
4. Select "Dashboard" Tab --> Click the plus icon --> Select "Visualization" tab --> Search for "Squid Event Count" --> Select it
5. The visualization will be added to the bottom of the dashboard
6. Click the save icon on the top right to save the dashboard.
   
   

Step 7: Creating a Histogram Panel

1. Log into the Metron UI Dashboard: http://METRON_UI_HOST:5000
2. Select "Visualize" Tab --> Select "Line Chart" Visualization --> Select "From a new search" for Search Source --> Select "squid*" index source 
3. Configure the Visualization like the following: 
   1. Image Added
      
4. Click the Save Icon on the right right corner --> Name the Visualization "Squid Events Histogram" and click Save
5. Select "Dashboard" Tab --> Click the plus icon --> Select "Visualization" tab --> Search for "Squid Events Histogram" --> Select it
6. The visualization will be added to the bottom of the dashboard
7. Click the save icon on the top right to save the dashboard.

Step 8: Adding a Detail Panel

1. Log into the Metron UI Dashboard: http://METRON_UI_HOST:5000
2. Select "Discover" Tab --> Select the "squid*" index
3. Search for only docs in this index with type of squid_doc
   1. Type the following in search "_type:  squid_doc" 
   2. click the search icon
4. Now we only to select subset of the fields that we want to display in the detail panel. In the left hand panel under "Available Fields", "add" the following fields:
   1. full_hostname
   2. ip_src_addr
   3. ip_dst_addr
   4. original_string
   5. method
   6. type
5. The discover/search panel should look something like the following:
   1. Image Added
6. Click the "Save" icon on the top right corner  --> name the search "Squid Event Details" --> Click Save
7. Select "Dashboard" Tab --> Click the plus icon --> Select "Searches" tab --> Search for "Squid Event Details" --> Select it
8. The visualization will be added to the bottom of the dashboard
9. Click the save icon on the top right to save the dashboard.

Step 9: The Dashboard with the 3 Squid Panels

The following is what the new dashboard would look like with the 3 squid panels added.

Image Added

Let's create a basic data table so that a user can inspect record-level details for Squid.  In Kibana, this is done by creating a 'Saved Search'

Image Removed

1. Click on `Discover` and then choose the newly created `squid*` index pattern.

2. In the 'Fields' panel on the left, choose which fields to include in the saved search.  Click the 'Add' button next to each field.

3. Click on the 'Save' icon near the top-right to save the search.

Step 7: Visualize the Squid Data

After using the `Discover` panel to better understand the Squid data, let's create a few visualizations.

Image Removed

1. Click on 'Visualize' in the top level menu.

2. Choose the 'Vertical bar chart' and when prompted to 'Select a search source' choose 'From a new search'. Choose the `squid*` index pattern.

3. Under 'Select bucket types' click the 'X-Axis' and for the 'Aggregation' type choose 'Terms'.

4. Under 'Field' choose the `domain_without_subdomains` field.

5. Click the 'Play' button to refresh the visualization.

6. Near the top-right side of the screen click on the 'Save' icon to save the visualization. Name it something appropriate. This will allow us to use the visualization in a dashboard later.

Step 7: Customize the Dashboard

Image Removed

1. Open the Metron Dashboard by clicking on 'Dashboard' in the top-level menu.

2. On the right, click the 'Add' button indicated by a plus sign.

3. Find the visualization that you would like to add.

4. Scroll to the bottom of the dashboard to find the visualization that was added. From here you can resize and move the visualization as needed.

5. Continue enhancing the dashboard by adding the 'Saved Search' that was previously created.

Summary

At this point you should be comfortable customizing a dashboard as you add new sources of telemetry to Metron. This article introduced Metron's default dashboard that is built upon Kibana 4. It covered the elements present in the dashboard and how you can extend the dashboard for your own purposes.