Versions Compared

Old Version 23

changes.mady.by.user George Vetticaden

Saved on

compared with

New Version 24

changes.mady.by.user George Vetticaden

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

  1. ssh into Host $HOST_WITH_ENRICHMENT_TAG as root
  2. Create a Squid Grok parser configuration file at /usr/metron/$METRON_VERSION/config/zookeeper/parsers/squid.json with the following contents: 

    {
     "parserClassName": "org.apache.metron.parsers.GrokParser",
     "sensorTopic": "squid",
     "parserConfig": {
     "grokPath": "/apps/metron/patterns/squid",
     "patternLabel": "SQUID_DELIMITED",
     "timestampField": "timestamp"
     },
     "fieldTransformations" : [
       {
         "transformation" : "MTL"
        ,"output" : [ "full_hostname", "domain_without_subdomains" ]
        ,"config" : {
                      "full_hostname" : "URL_TO_HOST(url)"
                     ,"domain_without_subdomains" : "DOMAIN_REMOVE_SUBDOMAINS(full_hostname)"
                    }
        }
      ]
     }

     

  3. Notice the use of the fieldTransformations in the parser configuration.  Our Grok Parser is set up to extract the URL, but really we want just the domain or even the domain without subdomains.  To do this, we can use the Metron Transformation Language field transformation.  The Metron Transformation Language is a Domain Specific Language which allows users to define extra transformations to be done on the messages flowing through the topology.  It supports a wide range of common network and string related functions as well as function composition and list operations.  In our case, we extract the hostname from the URL via the URL_TO_HOST function and remove the domain names with DOMAIN_REMOVE_SUBDOMAINS thereby creating two new fields, "full_hostname" and "domain_without_subdomains" to each message. 
  4. All parser configurations are stored in Zookeeper. A script is provided to upload configurations to Zookeeper. 
    1. /usr/metron/$METRON_VERSION/bin/zk_load_configs.sh --mode PUSH -i /usr/metron/$METRON_VERSION/config/zookeeper -z $ZOOKEEPER_HOST:2181 


 

Step 5: Validate the Squid Message

...

Another thing we can do is validate our messages.  Lets say we wanted to make sure that source IPs and destination IPs are valid.  The validators are global so we set them up on the global JSON and push them into Zookeeper.  The list of available validators can be found here: 
  1. ssh into Host $HOST_WITH_ENRICHMENT_TAG as root
  2. Open up the global validation configuration
    1. vi /usr/metron/$METRON_VERSION/config/zookeeper/global.json
    2. Add the following validation configuration to it and save it. 
      {
        "es.clustername": "metron",
        "es.ip": "$ES$SEARCH_CLUSTERHOST", //make sure to replace this
        "es.port": "9300$SEARCH_HOST_PORT", //make sure to replace this
        "es.date.format": "yyyy.MM.dd.HH",
        "fieldValidations" : [
         {
            "input" : [ "ip_src_addr", "ip_dst_addr" ],
            "validation" : "IP",
            "config" : {
            "type" : "IPV4"
             }
         }
        ]
       
      }

       

  3. Push the global configuration to zookeeper 
    1. /usr/metron/$METRON_VERSION/bin/zk_load_configs.sh -i /usr/metron/$METRON_VERSION/config/zookeeper -m PUSH -z $ZOOKEEPER_HOST:2181
  4. Dump the configs and validate it got persisted
    1. /usr/metron/$METRON_VERSION/bin/zk_load_configs.sh -m DUMP -z $ZOOKEEPER_HOST:2181

The below describes the validation configuration you see above.

More details on the validation framework can be found in the Validation Framework section here: https://github.com/apache/incubator-metron/tree/master/metron-platform/metron-common#transformation-language

 

Step 6: Deploy the new Parser Topology

...