Page History

...

• KAFKA_HOST = host where a Kafka broker is installed
• ZOOKEEPER_HOST = host where a Zookeeper server is installed
• PROBE_HOST = Host where your sensor, probes are installed. If don't have any sensors installed, pick the host where a storm supervisor is running
• SQUID_HOST = Host where you want to install SQUID. If you don't care, just install on the PROBE_HOST
• NIFI_HOST = The host where you will install NIFI. You want this this to be same host that you installed Squid.
• HOST_WITH_ENRICHMENT_TAG = This is the host in your inventory hosts file that you put under the group "enrichment" 
• HOST_WITH_SEARCH_TAG HOST = This is the host where you have elastic or solr running. This is the host in your inventory hosts file that you put under the group "search". Pick one of the search hosts
• SEARCH_HOST_PORT  = The port of the search host where indexing is configured. (e.g: 9300)
• METRON_UI_HOST HOST_WITH_WEB_TAG = This is the host where your metron ui web application is running. This is the host in your inventory hosts file that you put under the group "web".
• METRON_RELEASE VERSION = The release of the metron binaries you are working with (e.g: 0.2.0BETA-RC2)

...

1. ssh into Host $HOST_WITH_ENRICHMENT_TAG as root
2. Create a Squid Grok parser configuration file at /usr/metron/$METRON_RELEASEVERSION/config/zookeeper/parsers/squid.json with the following contents: 

   
   {
    "parserClassName": "org.apache.metron.parsers.GrokParser",
    "sensorTopic": "squid",
    "parserConfig": {
    "grokPath": "/apps/metron/patterns/squid",
    "patternLabel": "SQUID_DELIMITED",
    "timestampField": "timestamp"
    },
    "fieldTransformations" : [
      {
        "transformation" : "MTL"
       ,"output" : [ "full_hostname", "domain_without_subdomains" ]
       ,"config" : {
                     "full_hostname" : "URL_TO_HOST(url)"
                    ,"domain_without_subdomains" : "DOMAIN_REMOVE_SUBDOMAINS(full_hostname)"
                   }
       }
     ]

    }

    

3. Notice the use of the fieldTransformations in the parser configuration.  Our Grok Parser is set up to extract the URL, but really we want just the domain or even the domain without subdomains.  To do this, we can use the Metron Transformation Language field transformation.  The Metron Transformation Language is a Domain Specific Language which allows users to define extra transformations to be done on the messages flowing through the topology.  It supports a wide range of common network and string related functions as well as function composition and list operations.  In our case, we extract the hostname from the URL via the URL_TO_HOST function and remove the domain names with DOMAIN_REMOVE_SUBDOMAINS thereby creating two new fields, "full_hostname" and "domain_without_subdomains" to each message. 
4. All parser configurations are stored in Zookeeper. A script is provided to upload configurations to Zookeeper. 
   1. /usr/metron/$METRON_VERSION/bin/zk_load_configs.sh --mode PUSH -i /usr/metron/$METRON_VERSION/config/zookeeper -z $ZOOKEEPER:2181 

Step

...

5:

...

Validate the Squid Message

Step 6: Deploy the new Parser Topology

1. Log into HOST $HOST_WITH_ENRICHMENT_TAG as root
2. Deploy the new squid paser topology:

   /usr/metron/$METRON_RELEASEVERSION/bin/start_parser_topology.sh -k $KAFKA_HOST:6667 -z $ZOOKEEPER_HOST:2181 -s squid

3. Go to the Storm UI and you should now see new "squid" topology and ensure that the topology has no errors

...

In order to verify that the messages were indexed correctly, we can use the elastic search Head plugin.

1. ssh into Host $HOST_WITH_SEARCH_TAG$SEARCH_HOST
2. Install the head plugin

   usr/share/elasticsearch/bin/plugin -install mobz/elasticsearch-head/1.x 

3. Navigate to elastic head UI: http://HOST_WITH_SEARCH_TAGHOST:9200/_plugin/head/
4. Click on Browser tab and select squid doc on the left panel and then select one of th sample docs. You should see something like the following:
   1. 

...

1. Log into the Metron UI Dashboard: http://HOSTMETRON_WITHUI_WEB_TAGHOST:5000
2. Select "Visualize" Tab --> Select "Metric" Visualization"= --> Select "From a new search" for Search Source --> Select "squid*" index source –> Click the Save disk icon on the top right
3. Name the Visualization "Squid Event Count" and click Save
4. Select "Dashboard" Tab --> Click the plus icon --> Select "Visualization" tab --> Search for "Squid Event Count" --> Select it
5. The visualization will be added to the bottom of the dashboard
6. Click the save icon on the top right to save the dashboard.
   
   

...

1. Log into the Metron UI Dashboard: http://HOSTMETRON_WITHUI_WEB_TAGHOST:5000
2. Select "Visualize" Tab --> Select "Line Chart" Visualization --> Select "From a new search" for Search Source --> Select "squid*" index source 
3. Configure the Visualization like the following: 
   1. 
      
4. Click the Save Icon on the right right corner --> Name the Visualization "Squid Events Histogram" and click Save
5. Select "Dashboard" Tab --> Click the plus icon --> Select "Visualization" tab --> Search for "Squid Events Histogram" --> Select it
6. The visualization will be added to the bottom of the dashboard
7. Click the save icon on the top right to save the dashboard.

...

1. Log into the Metron UI Dashboard: http://HOSTMETRON_WITHUI_WEB_TAGHOST:5000
2. Select "Discover" Tab --> Select the "squid*" index
3. Search for only docs in this index with type of squid_doc
   1. Type the following in search "_type:  squid_doc" 
   2. click the search icon
4. Now we only to select subset of the fields that we want to display in the detail panel. In the left hand panel under "Available Fields", "add" the following fields:
   1. full_hostname
   2. ip_src_addr
   3. ip_dst_addr
   4. original_string
   5. method
   6. type
5. The discover/search panel should look something like the following:
   1. 
6. Click the "Save" icon on the top right corner  --> name the search "Squid Event Details" --> Click Save
7. Select "Dashboard" Tab --> Click the plus icon --> Select "Searches" tab --> Search for "Squid Event Details" --> Select it
8. The visualization will be added to the bottom of the dashboard
9. Click the save icon on the top right to save the dashboard.

...