Page History

Now that we have Metron configured to parse, index and persist telemetry events and Nifi pushing data to Metron, lets now visualize this streaming telemetry data in the Metron UI. We will be adding 3 new panels to visualize the Squid Events: Histogram Panel, Count Panel and Detail Panel

Step 1: Setup and

...

Prerequisites

1. Complete You should have completed the instructions in Adding a new Telemetry Data Source
2. Make sure the following variables are configured based on your environment: 

    

   • KAFKA_HOST = The host where a Kafka broker is installed.
   • ZOOKEEPER_HOST = The host where a Zookeeper server is installed.
   • PROBE_HOST =
   Host
   • The host where your sensor, probes are installed. If don't have any sensors installed, pick the host where a
   storm
   • Storm supervisor is running.
   • SQUID_HOST =
   Host
   • The host where you want to install SQUID. If you don't care, just install SQUID on the PROBE_HOST.
   • NIFI_HOST =
   The host
   • Host where you will install NIFI. You want this this to be same host
   that
   • on which you installed Squid.
   • HOST_WITH_ENRICHMENT_TAG =
   This is the
   • The host in your inventory hosts file that you put under the group "enrichment." 
   • SEARCH_
   HOST
   • HOST =
   This is the
   • The host where you have
   elastic
   • Elastic or
   solr
   • Solr running.
   This
   •  This is the host in your inventory hosts file that you put under the group "search". Pick one of the search hosts.
   • SEARCH_HOST_PORT  = The port of the search host where indexing is configured. (e.g
   :
   • ., 9300)
   • METRON_UI_HOST =
   This is the
   • The host where your
   metron ui
   • Metron UI web application is running.
   This
   •  This is the host in your inventory hosts file that you put under the group "web."
   .
   • METRON_VERSION = The release of the
   metron
   • Metron binaries you are working with. (e.g
   :
   • ., 0.2.0BETA-RC2)

Step 2: Create More Squid Sensor Data

...

1. Run the following command to create an index template for Squid. 

    curl -XPOST $SEARCH_HOST:$SEARCH_PORT/_template/squid_index -d '
    {
    "template": "squid_index*",
    "mappings": {
    "brosquid_doc": {
    "_timestamp": {
    "enabled": true
    },
    "properties": {
    "timestamp": {
    "type": "date",
    "format": "epoch_millis"
    },
    "source:type": {
    "type": "string",
    "index": "not_analyzed"
    },
    "action": {
    "type": "string",
    "index": "not_analyzed"
    },
    "bytes": {
    "type": "integer"
    },
    "code": {
    "type": "string",
    "index": "not_analyzed"
    },
    "domain_without_subdomains": {
    "type": "string",
    "index": "not_analyzed"
    },
    "full_hostname": {
    "type": "string",
    "index": "not_analyzed"
    },
    "elapsed": {
    "type": "integer"
    },
    "method": {
    "type": "string",
    "index": "not_analyzed"
    },
    "ip_dst_addr": {
    "type": "string",
    "index": "not_analyzed"
    }
    }
    }
    }
    }'

2. By default, Elasticsearch will attempt to analyze all fields of type string. This means that Elasticsearch will tokenize the string and perform additional processing to enable free-form text search. In many cases, and all cases for the Squid data, we want to treat each of the string fields as enumerations. This is why most fields in the index template are `not_analyzed`.
3. An index template will only apply for indices that are created after the template is created. Delete the existing Squid indices so that new ones can be generated with the index template. 

   curl -XDELETE $SEARCH_HOST:9200/squid*

4. Wait for the Squid index to be re-created. This may take a minute or two based on how fast the Squid data is being consumed in your environment. 

   curl -XGET node1:9200/squid*

...