Versions Compared

Old Version 2

changes.mady.by.user George Vetticaden

Saved on

compared with

New Version Current

changes.mady.by.user Otto Fowler

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: fix reference to bro_doc

Now that we have Metron configured to parse, index and persist telemetry events and Nifi pushing data to Metron, lets now visualize this streaming telemetry data in the Metron UI. We will be adding 3 new panels to visualize the Squid Events: Histogram Panel, Count Panel and Detail Panel

 

Table of Contents

Step 1: Setup and Prerequisites

  1. Complete the instructions in Adding a new Telemetry Data Source
  2. Make sure the following variables are configured based on your environment: 

     

    • KAFKA_HOST = The host where a Kafka broker is installed.
    • ZOOKEEPER_HOST = The host where a Zookeeper server is installed.
    • PROBE_HOST = The host where your sensor, probes are installed. If don't have any sensors installed, pick the host where a Storm supervisor is running.
    • SQUID_HOST = The host where you want to install SQUID. If you don't care, just install SQUID on the PROBE_HOST.
    • NIFI_HOST = Host where you will install NIFI. You want this this to be same host on which you installed Squid.
    • HOST_WITH_ENRICHMENT_TAG = The host in your inventory hosts file that you put under the group "enrichment." 
    • SEARCH_HOST = The host where you have Elastic or Solr running. This is the host in your inventory hosts file that you put under the group "search". Pick one of the search hosts.
    • SEARCH_HOST_PORT  = The port of the search host where indexing is configured. (e.g., 9300)
    • METRON_UI_HOST = The host where your Metron UI web application is running. This is the host in your inventory hosts file that you put under the group "web."
    • METRON_VERSION = The release of the Metron binaries you are working with. (e.g., 0.2.0BETA-RC2)

Step 2: Create More Squid Sensor

Metron's Dashboard

Metron's default dashboard is intended to allow you to easily validate the end-to-end functioning of Metron with its default sensor suite. It highlights some of the useful widgets available in Kibana 4, and serves as a starting point for you to build your own customized dashboards.

Image Removed

The first panel in the dashboard highlights the variety of events being consumed by Metron. It shows the total number of events received, the variety of those events, and a histogram showing when the events were received.

Image Removed

The next set of dashboard panels shows how Apache Metron can be used to perform real-time enrichment of telemetry data. All of the IPv4 data received by Metron was cross-referenced against a geo-ip database. These locations were then used to build this set of dashboard widgets.

Image Removed

 

As part of the default sensor suite, YAF is used to generate flow records. These flow records provide significant visibility into which actors are communicating over the target network. A table widget displays the raw details of each flow record. A histogram of the duration of each flow shows that while most flows are relatively short-lived there are a few that are exceptionally longer in this example. Creating an index template that defined this field as numeric was required to generate the histogram.

Image Removed

Snort is a Network Intrusion Detection System (NIDS) that is being used to generate alerts identifying known bad events. Snort relies on a fixed set of rules that act as signatures for identifying abnormal events. Along with displaying the relevant details of each alert, the panel shows that there is only a single unique alert type; a test rule that creates a Snort alert on every network packet. Another table was created to show source/destination pairs that generated the most Snort alerts.

Image Removed

The Bro Network Security Monitor is extracting application-level information from raw network packets. In this example, Bro is extracting HTTP and HTTPS requests being made over the network. The panels highlight the breakdown by request type, the total number of web requests, and raw details from each web request.

Image Removed

Bro is extracting DNS requests and responses being made over the network. Understanding who is making those requests, the frequency, and types can provide a deep understanding of the actors present on the network.

Creating Your Own Dashboard

Now that you understand Metron's default dashboard, let's cover how you might extend this dashboard for your own purposes. We will continue the ongoing example of parsing Squid Proxy logs. The dashboard will be extended to display the Squid log data.

...

Data

The previous tutorials covering Squid produced a limited data set. These consisted of a few basic requests. To make this tutorial more interesting, we are going to need a bit more variety in the sample data.1.

  1. ssh into SQUID_HOST as root
  2. Copy and paste the following set of links to a local file called `links.txt`.  
     

...

  1. https://www.amazon.com/Cards-Against-Humanity-LLC-CAHUS/dp/B004S8F7QM/ref=zg_bs_toys-and-games_home_1?pf_rd_p=2140216822&pf_rd_s=center-1&pf_rd_t=2101&pf_rd_i=home&pf_rd_m=ATVPDKIKX0DER&pf_rd_r=2231TS0FE044EZT85PQ4
        https://www.amazon.com/Brain-Game-Cube-Intelligence-Development/dp/B01CRXM1JU/ref=zg_bs_toys-and-games_home_2?pf_rd_p=2140216822&pf_rd_s=center-1&pf_rd_t=2101&pf_rd_i=home&pf_rd_m=ATVPDKIKX0DER&pf_rd_r=MANXEWDTKDH2RD9Y3466
        https://www.amazon.com/Zuru-Balloons-different-colors-Seconds/dp/B00ZPW3U14/ref=zg_bs_toys-and-games_home_3?pf_rd_p=2140216822&pf_rd_s=center-1&pf_rd_t=2101&pf_rd_i=home&pf_rd_m=ATVPDKIKX0DER&pf_rd_r=MANXEWDTKDH2RD9Y3466
        https://www.amazon.com/MAGINOVO-Bluetooth-Headphones-Wireless-Earphones/dp/B01EFKFQL8/ref=zg_bs_electronics_home_1?pf_rd_p=2140225402&pf_rd_s=center-2&pf_rd_t=2101&pf_rd_i=home&pf_rd_m=ATVPDKIKX0DER&pf_rd_r=MANXEWDTKDH2RD9Y3466
        https://www.amazon.com/Amazon-Fire-TV-Stick-Streaming-Media-Player/dp/B00GDQ0RMG/ref=zg_bs_electronics_home_2?pf_rd_p=2140225402&pf_rd_s=center-2&pf_rd_t=2101&pf_rd_i=home&pf_rd_m=ATVPDKIKX0DER&pf_rd_r=MANXEWDTKDH2RD9Y3466
        http://www.walmart.com/ip/All-the-Light-We-Cannot-See/26737727
        http://www.walmart.com/ip/Being-Mortal-Medicine-and-What-Matters-in-the-End/36958209
        http://www.walmart.com/ip/My-Brilliant-Friend-Book-One-Childhood-Adolescence/20527482
        http://www.walmart.com/ip/A-Game-of-Thrones/402949
        http://www.bbc.co.uk/capital/story/20160622-there-are-people-making-millions-from-your-pets-poo
        http://www.bbc.co.uk/earth/story/20160620-can-we-predict-the-time-of-our-death
        http://www.bbc.co.uk/news/uk-england-somerset-36596557

...


  2.  Run this command to choose one of the links above at random and make a request for that link through Squid. Leave this command running in a terminal so that a continual feed of data is generated as we work through the remainder of this tutorial. 
...
  1.       
...
  1.  while sleep 2; do cat links.txt | shuf -n 1 | xargs -i squidclient -g 4 -v {}; done
...
  1. The previous command is generating log records at `/var/log/squid/access.log`. 
...
  1. As long as Nifi is still running that we created in Adding a new Telemetry Data Source these event should be pushed to Metron's Telemetry Ingest layer
  1. . Ensure that the parser topology for Squid continues to run based on the steps outlined in the previous tutorials.
Step 3: Create an Index Template
To work with the Squid data in Kibana, we need to ensure that the data is landing in the search index with the correct data types. This can be achieved by defining an index template.1. 
  1. Run the following command to create an index template for Squid. 
     curl -XPOST $
...
  1. SEARCH_HOST:
...
  1. $SEARCH_PORT/_template/squid_index -d '
     {
     "template": "squid_index*",
     "mappings": {
     "
...
  1. squid_doc": {
     "_timestamp": {
     "enabled": true
     },
     "properties": {
     "timestamp": {
     "type": "date",
     "format": "epoch_millis"
     },
     "source:type": {
     "type": "string",
     "index": "not_analyzed"
     },
     "action": {
     "type": "string",
     "index": "not_analyzed"
     },
     "bytes": {
     "type": "integer"
     },
     "code": {
     "type": "string",
     "index": "not_analyzed"
     },
     "domain_without_subdomains": {
     "type": "string",
     "index": "not_analyzed"
     },
     "full_hostname": {
     "type": "string",
     "index": "not_analyzed"
     },
     "elapsed": {
     "type": "integer"
     },
     "method": {
     "type": "string",
     "index": "not_analyzed"
     },
     "ip_dst_addr": {
     "type": "string",
     "index": "not_analyzed"
     }
     }
     }
     }
     }'
...
  1. By default, Elasticsearch will attempt to analyze all fields of type string. This means that Elasticsearch will tokenize the string and perform additional processing to enable free-form text search. In many cases, and all cases for the Squid data, we want to treat each of the string fields as enumerations. This is why most fields in the index template are `not_analyzed`.
...
  1. An index template will only apply for indices that are created after the template is created. Delete the existing Squid indices so that new ones can be generated with the index template. 
...
  1. curl -XDELETE 
...
  1. $SEARCH_HOST:9200/squid*
...
  1. Wait for the Squid index to be re-created. This may take a minute or two based on how fast the Squid data is being consumed in your environment. 
...
  1. curl -XGET node1:9200/squid*
Step 4: Configure the Squid Index in Kibana
Now that we have a Squid index with all of the right data types, we need to tell Kibana about this index.
 Info
 
Click on the image above to see each of these steps performed.
...
1. Login to your Kibana user interface and http://METRON_UI_HOST:5000 and then click on 'Settings', then 'Indices'.
...
4. Then click the 'Create' button.
Step 5: Review the Squid Data
Now that Kibana is aware of the new Squid index, let's take a look at the data.
...
3. Clicking on a specific record will show each field available in the data.
Save a Squid Search
Let's create a basic data table so that a user can inspect record-level details for Squid.  In Kibana, this is done by creating a 'Saved Search'
 
Image Removed
 Info
Click on the image above to see each of these steps performed.
 
1. Click on `Discover` and then choose the newly created `squid*` index pattern.
2. In the 'Fields' panel on the left, choose which fields to include in the saved search.  Click the 'Add' button next to each field.
3. Click on the 'Save' icon near the top-right to save the search.
Visualize the Squid Data
After using the `Discover` panel to better understand the Squid data, let's create a few visualizations.
Image Removed
 Info
Click on the image above to see each of these steps performed.
 
1. Click on 'Visualize' in the top level menu.
2. Choose the 'Vertical bar chart' and when prompted to 'Select a search source' choose 'From a new search'. Choose the `squid*` index pattern.
3. Under 'Select bucket types' click the 'X-Axis' and for the 'Aggregation' type choose 'Terms'.
4. Under 'Field' choose the `domain_without_subdomains` field.
5. Click the 'Play' button to refresh the visualization.
6. Near the top-right side of the screen click on the 'Save' icon to save the visualization. Name it something appropriate. This will allow us to use the visualization in a dashboard later.
Customize the Dashboard
Image Removed
 Info
Click on the image above to see each of these steps performed.
 
1. Open the Metron Dashboard by clicking on 'Dashboard' in the top-level menu.
2. On the right, click the 'Add' button indicated by a plus sign.
3. Find the visualization that you would like to add.
4. Scroll to the bottom of the dashboard to find the visualization that was added. From here you can resize and move the visualization as needed.
Step 6: Adding Squid Event Count Panel to Dashboard
  1. Log into the Metron UI Dashboard: http://METRON_UI_HOST:5000
  2. Select "Visualize" Tab --> Select "Metric" Visualization"= --> Select "From a new search" for Search Source --> Select "squid*" index source –> Click the Save disk icon on the top right
  3. Name the Visualization "Squid Event Count" and click Save
  4. Select "Dashboard" Tab --> Click the plus icon --> Select "Visualization" tab --> Search for "Squid Event Count" --> Select it
  5. The visualization will be added to the bottom of the dashboard
  6. Click the save icon on the top right to save the dashboard.

Step 7: Creating a Histogram Panel
  1. Log into the Metron UI Dashboard: http://METRON_UI_HOST:5000
  2. Select "Visualize" Tab --> Select "Line Chart" Visualization --> Select "From a new search" for Search Source --> Select "squid*" index source 
  3. Configure the Visualization like the following: 
    1. Image Added
  4. Click the Save Icon on the right right corner --> Name the Visualization "Squid Events Histogram" and click Save
  5. Select "Dashboard" Tab --> Click the plus icon --> Select "Visualization" tab --> Search for "Squid Events Histogram" --> Select it
  6. The visualization will be added to the bottom of the dashboard
  7. Click the save icon on the top right to save the dashboard.
Step 8: Adding a Detail Panel
  1. Log into the Metron UI Dashboard: http://METRON_UI_HOST:5000
  2. Select "Discover" Tab --> Select the "squid*" index
  3. Search for only docs in this index with type of squid_doc
    1. Type the following in search "_type:  squid_doc" 
    2. click the search icon
  4. Now we only to select subset of the fields that we want to display in the detail panel. In the left hand panel under "Available Fields", "add" the following fields:
    1. full_hostname
    2. ip_src_addr
    3. ip_dst_addr
    4. original_string
    5. method
    6. type
  5. The discover/search panel should look something like the following:
    1. Image Added
  6. Click the "Save" icon on the top right corner  --> name the search "Squid Event Details" --> Click Save
  7. Select "Dashboard" Tab --> Click the plus icon --> Select "Searches" tab --> Search for "Squid Event Details" --> Select it
  8. The visualization will be added to the bottom of the dashboard
  9. Click the save icon on the top right to save the dashboard.
Step 9: The Dashboard with the 3 Squid Panels
The following is what the new dashboard would look like with the 3 squid panels added.
Image Added5. Continue enhancing the dashboard by adding the 'Saved Search' that was previously created.
Summary
At this point you should be comfortable customizing a dashboard as you add new sources of telemetry to Metron. This article introduced Metron's default dashboard that is built upon Kibana 4. It covered the elements present in the dashboard and how you can extend the dashboard for your own purposes.