Page History

...

Now that you understand Metron's default dashboard, let's cover how you might extend this dashboard for your own purposes. We will continue the ongoing example of parsing Squid Proxy logs. The dashboard will be extended to display the Squid log data.

...

Create More Squid Data

The previous tutorials covering Squid produced a limited data set. These consisted of a few basic requests. To make this tutorial more interesting, we are going to need a bit more variety in the sample data.1.

1. ssh into SQUID_HOST as root
2. Copy and paste the following set of links to a local file called `links.txt`.  

...

1. https://www.amazon.com/Cards-Against-Humanity-LLC-CAHUS/dp/B004S8F7QM/ref=zg_bs_toys-and-games_home_1?pf_rd_p=2140216822&pf_rd_s=center-1&pf_rd_t=2101&pf_rd_i=home&pf_rd_m=ATVPDKIKX0DER&pf_rd_r=2231TS0FE044EZT85PQ4
       https://www.amazon.com/Brain-Game-Cube-Intelligence-Development/dp/B01CRXM1JU/ref=zg_bs_toys-and-games_home_2?pf_rd_p=2140216822&pf_rd_s=center-1&pf_rd_t=2101&pf_rd_i=home&pf_rd_m=ATVPDKIKX0DER&pf_rd_r=MANXEWDTKDH2RD9Y3466
       https://www.amazon.com/Zuru-Balloons-different-colors-Seconds/dp/B00ZPW3U14/ref=zg_bs_toys-and-games_home_3?pf_rd_p=2140216822&pf_rd_s=center-1&pf_rd_t=2101&pf_rd_i=home&pf_rd_m=ATVPDKIKX0DER&pf_rd_r=MANXEWDTKDH2RD9Y3466
       https://www.amazon.com/MAGINOVO-Bluetooth-Headphones-Wireless-Earphones/dp/B01EFKFQL8/ref=zg_bs_electronics_home_1?pf_rd_p=2140225402&pf_rd_s=center-2&pf_rd_t=2101&pf_rd_i=home&pf_rd_m=ATVPDKIKX0DER&pf_rd_r=MANXEWDTKDH2RD9Y3466
       https://www.amazon.com/Amazon-Fire-TV-Stick-Streaming-Media-Player/dp/B00GDQ0RMG/ref=zg_bs_electronics_home_2?pf_rd_p=2140225402&pf_rd_s=center-2&pf_rd_t=2101&pf_rd_i=home&pf_rd_m=ATVPDKIKX0DER&pf_rd_r=MANXEWDTKDH2RD9Y3466
       http://www.walmart.com/ip/All-the-Light-We-Cannot-See/26737727
       http://www.walmart.com/ip/Being-Mortal-Medicine-and-What-Matters-in-the-End/36958209
       http://www.walmart.com/ip/My-Brilliant-Friend-Book-One-Childhood-Adolescence/20527482
       http://www.walmart.com/ip/A-Game-of-Thrones/402949
       http://www.bbc.co.uk/capital/story/20160622-there-are-people-making-millions-from-your-pets-poo
       http://www.bbc.co.uk/earth/story/20160620-can-we-predict-the-time-of-our-death
       http://www.bbc.co.uk/news/uk-england-somerset-36596557
   

...

2.  Run this command to choose one of the links above at random and make a request for that link through Squid. Leave this command running in a terminal so that a continual feed of data is generated as we work through the remainder of this tutorial. 

...

...

1.  while sleep 2; do cat links.txt | shuf -n 1 | xargs -i squidclient -g 4 -v {}; done

...

1. The previous command is generating log records at `/var/log/squid/access.log`.

...

1. As long as Nifi is still running that we created in Adding a new Telemetry Data Source these event should be pushed to Metron's Telemetry Ingest layer
2. .

...

1. Ensure that the parser topology for Squid continues to run based on the steps outlined in the previous tutorials.

Create an Index Template

To work with the Squid data in Kibana, we need to ensure that the data is landing in the search index with the correct data types. This can be achieved by defining an index template.1.

1. Run the following command to create an index template for Squid. 

    curl -XPOST $

...

1. SEARCH_HOST:

...

1. $SEARCH_PORT/_template/squid_index -d '
    {
    "template": "squid_index*",
    "mappings": {
    "bro_doc": {
    "_timestamp": {
    "enabled": true
    },
    "properties": {
    "timestamp": {
    "type": "date",
    "format": "epoch_millis"
    },
    "source:type": {
    "type": "string",
    "index": "not_analyzed"
    },
    "action": {
    "type": "string",
    "index": "not_analyzed"
    },
    "bytes": {
    "type": "integer"
    },
    "code": {
    "type": "string",
    "index": "not_analyzed"
    },
    "domain_without_subdomains": {
    "type": "string",
    "index": "not_analyzed"
    },
    "full_hostname": {
    "type": "string",
    "index": "not_analyzed"
    },
    "elapsed": {
    "type": "integer"
    },
    "method": {
    "type": "string",
    "index": "not_analyzed"
    },
    "ip_dst_addr": {
    "type": "string",
    "index": "not_analyzed"
    }
    }
    }
    }
    }'

...

1. By default, Elasticsearch will attempt to analyze all fields of type string. This means that Elasticsearch will tokenize the string and perform additional processing to enable free-form text search. In many cases, and all cases for the Squid data, we want to treat each of the string fields as enumerations. This is why most fields in the index template are `not_analyzed`.

...

1. An index template will only apply for indices that are created after the template is created. Delete the existing Squid indices so that new ones can be generated with the index template. 

...

1. curl -XDELETE 

...

1. $SEARCH_HOST:9200/squid*

...

1. Wait for the Squid index to be re-created. This may take a minute or two based on how fast the Squid data is being consumed in your environment. 

...

1. curl -XGET node1:9200/squid*

Configure the Squid Index in Kibana

...