Page History

...

1. The first thing we need to do is decide if we will be using the Java-based parser or the Grok-based parser for the new telemetry. In this example we will be using the Grok parser. Grok parser is perfect for structured or semi-structured logs that are well understood (check) and telemetries with lower volumes of traffic (check).
2. Next we need to define the Grok expression for our log. Refer to Grok documentation for additional details. In our case the pattern is:

   SQUID_DELIMITED %{NUMBER:timestamp}%{SPACE:UNWANTED} %{INT:elapsed}%{SPACE:UNWANTED}%{IPV4:ip_src_addr} %{WORD:action}/%{NUMBER:code} %{NUMBER:bytes} %{WORD:method} %{NOTSPACE:url} - %{WORD:UNWANTED}\/%{IPV4:ip_dst_addr} %{WORD:UNWANTED}\/%{WORD:UNWANTED}

   Notice that we apply the UNWANTED tag for any part of the message that we don't want included in our resulting JSON structure. Finally, notice that we applied the naming convention to the IPV4 field by referencing the following list of field conventions. 

3. The last thing we need to do is to validate the Grok pattern to make sure it's valid. For our test we will be using a free Grok validator called Grok Constructor. A validated Grok expression should look like this:

   

   

4. Now that the Grok pattern has been defined, we need to save it and move it to HDFS. 
   1. ssh into HOST $HOST_WITH_ENRICHMENT_TAG as root.

   2. Create a file called "squid" in the tmp directory and copy the Grok pattern into the file.

      touch /tmp/squid

   3. Open the Squid file and add the Grok pattern defined above.

   4. Put

      the

      Squid

      file

      into

      the

      directory

      where

      Metron

      stores

      its

      Grok

      parsers.

      Existing

      Grok

      parsers

      that

      ship

      with

      Metron

      are

      staged

      under

      /apps/metron/pattern.
      su

      -

      hdfs

      hadoop fs -rm -r /apps/metron/patterns/squid

      hdfs dfs -put /tmp/squid /apps/metron/patterns/

...