Page History

...

In the previous section, we walked through how to add a new Squid data source to Apache Metron. The inevitable next question is how I can I enrich the telemetry events in real-time as it flows they flow through the platform? Enrichment is critical when identifying threats or as we like to call it "finding the needle in the haystack." . The customers requirements are the following:

1. The proxy events from the Squid logs must be ingested in real-time.
2. The proxy logs must be parsed into a standardized JSON structure that Metron can understand.
3. In real-time, the squid proxy event must be enriched so that the domain names are enriched with the IP information.
4. In real-time, the IP with in within the proxy event must be checked against for threat intel feeds.
5. If there is a threat intel hit, an alert must be raised.
6. The end user must be able to see the new telemetry events and the alerts from the new data source.
7. All of these requirements must be implemented easily without writing any new Java code.

...

1. Configure an extractor config file that describes the enrichment source.  Cut and paste this file into a file called "extractor_config_temp.json."        

   {
   "config" : {
       "columns" : {
           "domain" : 0
           ,"owner" : 1
           ,"home_country" : 2
           ,"registrar": 3
           ,"domain_created_timestamp": 4
       }
       ,"indicator_column" : "domain"
       ,"type" : "whois"
       ,"separator" : ","
     }
     ,"extractor" : "CSV"
   }

2. Because copying and pasting from this blog will include some non-ascii invisible characters, run the following command to strip them out:

   1. iconv -c -f utf-8 -t ascii extractor_config_temp.json -o extractor_config.json

       

3. We will use the extractor_config file in step 4

...

1. As root user, log $HOST_WITH_ENRICHMENT_TAG.
2. Cut and paste the following into file into a file called "enrichment_config_temp.json" (make sure to set ZOOKEEPER_HOST with your specific value)  

   {
        "zkQuorum" : "$ZOOKEEPER_HOST:2181"
       ,"sensorToFieldList" : {
             "squid" : {
                "type" : "ENRICHMENT"
               ,"fieldToEnrichmentTypes" : {
                    "domain_without_subdomains" : [ "whois" ]
                 }
             }
       }
   }

3. Because copying and pasting from this blog will include some non-ascii invisible characters, run the following command to strip them out, ru the following:

   iconv -c -f utf-8 -t ascii enrichment_config_temp.json -o enrichment_config.json

   
   We will use the extractor_config file in step 5.

Step 5: Run the Enrichment Loader

1. Now that we have the enrichment source and enrichment config defined, we can now run the loader to move the data from the enrichment source to the Metron enrichment Store and store the enrichment config in Zookeeper.

   /usr/metron/$METRON_RELEASE/bin/flatfile_loader.sh -n enrichment_config.json -i whois_ref.csv -t enrichment -c t -e extractor_config.json

2. This command loads your enrichment data in Hbase and establishes a Zookeeper mapping. The data is populated into an Hbase table called enrichment. To verify that the logs were properly ingested into Hbase, run the following command: 
   

   hbase shell
   scan 'enrichment'

3. To check if Zookeeper enrichment tag was properly populated, run the following:

   /usr/metron/0.1BETA/bin/zk_load_configs.sh -m DUMP -z ZOOKEEPER_HOST:2181

4. Generate some data by using the Squid client to execute http requests. (Do this about 20 times.)

   squidclient http://www.cnn.com

Step

...

6: View the New Enriched Telemetry Events in Metron UI

1. Go to the Metron UI: http://METRON_UI_HOST:5000.
2. Select the Dashboard Tab.
3. Edit the Squid Event Details Panel that you created in the Add Telemetry Docs procedure by clicking on the edit icon. Metron displays the Discover page. 
4. Add the following new enrichment fields to the selected fields section (see the section highlighted in red):
   Image Modified
5. Click the Save Button to save the Search and ; save it with same name "Squid Event Details". 
6. Click on the Dashboard Page and delete the Squid Event Details panel, then re-add it. 
   The Squid Event Details panel should now have the new enriched fields.
    
    

...