Table of Contents

In the previous article of the sereiesseries, Enriching Telemetry Events, we walked through how to enrich a domain element of a given telemetry event with WhoIs data like home country, company associated with domain, etc. In this article, we will enrich with a special type of data called threat intel feeds. When a given telemetry event matches data in a threat Intel feed, the system generates an alert is generated.

Again, the customers requirement are the following:

1. The proxy events from Squid logs needs to must be ingested in real-time.
2. The proxy logs has to must be parsed into a standardized JSON structure that Metron can understand.
3. In real-time, the squid proxy event needs to must be enriched so that the domain named are enriched with the IP information
4. In real-time, the IP with in the proxy event must be checked against for threat intel feeds.
5. If there is a threat intel hit, an alert needs to must be raised.
6. The end user must be able to see the new telemetry events and the alerts from the new data source.
7. All of this requirements will need to must be implemented easily without writing any new java Java code.

In this article, we will walk you through how to do meet requirements 4 and 5.

Threat Intel Framework Explained

Metron currently provides an extensible framework to plug in threat intel sources. Each threat intel source has two components: an enrichment data source and and enrichment bolt. The threat intelligence feeds are bulk loaded and streamed into a threat intelligence store similarly similar to how the enrichment feeds are loaded. The keys are loaded in a key-value format. The key is the indicator and the value is the JSON formatted description of what the indicator is. It is recommended to use We recommend using a threat feed aggregator such as Soltra to dedup and normalize the feeds via Stix/Taxii. Metron provides an adapter that is able to read Soltra-produced Stix/Taxii feeds and stream them into HbaseHBase, which is the preferred data store of choice to back high-speed threat intel lookups of on Metron. Metron additionally provides a flat file and Stix bulk loader that can normalize, dedup, and bulk load or stream threat intel data into Hbase HBase even without the use of a threat feed aggregator.

The below following diagram illustrates the architecture:

Step 1: Setup and

...

Prerequisites

1. Complete You should have completed the instructions in Adding a new Telemetry Data Source.

2. Make sure the following variables are configured based on your environment: 

    

   • KAFKA_HOST = The host where a Kafka broker is installed.
   • ZOOKEEPER_HOST = The host where a Zookeeper server is installed.
   • PROBE_HOST =
   Host
   • The host where your sensor, probes are installed. If don't have any sensors installed, pick the host where a
   storm
   • Storm supervisor is running.
   • SQUID_HOST =
   Host
   • The host where you want to install SQUID. If you don't care, just install SQUID on the PROBE_HOST.
   • NIFI_HOST =
   The host
   • Host where you will install NIFI. You want this this to be same host
   that
   • on which you installed Squid.
   • HOST_WITH_ENRICHMENT_TAG =
   This is the
   • The host in your inventory hosts file that you put under the group "enrichment." 
   • SEARCH_
   HOST
   • HOST =
   This is the
   • The host where you have
   elastic
   • Elastic or
   solr
   • Solr running.
   This
   •  This is the host in your inventory hosts file that you put under the group "search". Pick one of the search hosts.
   • SEARCH_HOST_PORT  = The port of the search host where indexing is configured. (e.g
   :
   • ., 9300)
   • METRON_UI_HOST =
   This is the
   • The host where your
   metron ui
   • Metron UI web application is running.
   This
   •  This is the host in your inventory hosts file that you put under the group "web."
   .
   • METRON_VERSION = The release of the
   metron
   • Metron binaries you are working with. (e.g
   :
   • ., 0.2.0BETA-RC2)

Step 2: Create a Mock Threat Intel Feed Source

Metron is designed to work with Stix/Taxii threat feeds, but can also be bulk loaded with threat data from a CSV file. In this example, we will explore the CSV example. The same loader framework that is used for enrichment here is used for threat intelligence. Similarly Similar to enrichments, we need to setup set up a data.csv file, the extractor config JSON, and the enrichment config JSON.

For this example, we will be using use a Zeus malware tracker list located here: https://zeustracker.abuse.ch/blocklist.php?download=domainblocklist.

1. Log into the $HOST_WITH_ENRICHMENT_TAG as root.
2. Let's copy the contents from that link to a
3. Copy the data form the above link into a file called domainblocklist.txt on your VM.
4. Run the following command to parse the above file to a csv file called domainblocklist.csv.

   cat domainblocklist.txt
   curl https://zeustracker.abuse.ch/blocklist.php?download=domainblocklist | grep -v "^#" | grep -v "^$" | grep -v "^https" | awk '{print $1",abuse.ch”ch"}' > domainblocklist.csv

Step 3: Configure an Extractor Config File

1. Log into the $HOST_WITH_ENRICHMENT_TAG as root.
2. Now that we have the "Threat Intel Feed Source," , we need to now configure an extractor config file that describes the the source. Create a file called extractor_config_temp.json and put add the following contents in content to it. 

   
   {

   
    "config" : {

   
       "columns" : {

   
           "domain" : 0

   
           ,"source" : 1

   
       }

   
       ,"indicator_column" : "domain"

   
       ,"type" : "zeusList"

   
       ,"separator" : ","

   
     }

   
     ,"extractor" : "CSV"

   
   }

    

3. Run the following command to remove the non-ascii characters we run the following:
   

   iconv -c -f utf-8 -t ascii extractor_config_temp.json -o extractor_config.json

Step 4: Configure Element to Threat Intel Feed Mapping

We now have need to configure what which element of a tuple and what which threat intel feed with which to cross-reference with.This configuration will be stored in zookeeper.

The config looks like the following:

...

Zookeeper.

1. Log into the $HOST_WITH_ENRICHMENT_TAG as root.
2. Cut and paste this file into a file called "enrichment_config_temp.json." 

   
   {
        "zkQuorum" : "

...

1. $ZOOKEEPER_HOST:2181"

...

1. 
       ,"sensorToFieldList" : {

...

...

1.  "

...

1. squid" : {

...

...

1. "type" : "THREAT_INTEL"

...

...

1.    ,"fieldToEnrichmentTypes" : {

...

...

1.    "domain_without_subdomains" : [ "zeusList" ]

...

...

1.     }
        }
      }
   }

1. Because copying and pasting from this blog will include some non-ascii invisible characters, run the following command to strip them out

...

1. :

   iconv -c -f utf-8 -t ascii enrichment_config_temp.json -o enrichment_config.json

Step 5: Run the Threat Intel Loader

Now that we have the threat intel source, threat intel exractor, and threat intel mapping config defined, we can now run the loader to move the data from the threat intel source to the Metron threat intel Store and store the enrichment config in zookeeperZookeeper.

1. Log into the $HOST_WITH_ENRICHMENT_TAG as root.
2. Run the loader.

   /usr/metron/

...

1. $METRON_RELEASE/bin/flatfile_loader.sh -n enrichment_config.json -i 

...

1. domainblocklist.csv -t threatintel -c t -e extractor_config.json

...

2. The previous command adds

...

1. the threat intel data

...

1. into HBase and establishes a Zookeeper mapping

...

1. . The data

...

1. is populated into

...

1. an HBase table called threatintel. To verify that the logs were properly ingested into

...

1. HBase, run the following command: 

   hbase shell

...

1. 
   scan 'threatintel'

...

...

1. Now check if the Zookeeper enrichment tag was properly populated:  

   /usr/metron/

...

1. $METRON_RELEASE/bin/zk_load_configs.sh -m DUMP -z 

...

1. $ZOOKEEPER_HOST:2181

2. You should see a config for the Squid sensor something like the following: 
   Image Added
3. Generate some data by using the

...

1. Squid client to execute http requests. (

...

1. Do this about 20 times.)

   squidclient http://www.

...

1. actdhaka.com

...

Step

...

6: View the Threat Alerts in Metron UI

When the logs are ingested we get messages that has a hit against threat intel:

Image Removed

Notice a couple of characteristics about this message. It has is_alert=true, which designates it as an alert message.

Now that we have alerts coming through we need to visualize them in Kibana. First, we need to setup a pinned query to look for messages where is_alert=true:

Image Removed

And then once we point the alerts table to this pinned query it looks like this:

Now that we have configured real-time threat intel cross referencing so that alerts get generated when there is a hit for the Squid sensor, let's render these alerts on the Metron UI. We will be adding two new panels to visualize the Squid Alerts.

Creating a Threat Intel Hits Count Panel 

1. Log into the Metron UI Dashboard: http://METRON_UI_HOST:5000.
2. Select "Visualize" Tab --> Select "Metric" Visualization"= --> Select "From a new search" for Search Source --> Select "squid*" index source.
3. In the search box, enter "is_alert =  true" then execute the search.
4. Click the Save disk icon on the top right and name the Visualization "Threat Intel Hits," then click Save.
5. Select "Dashboard" Tab --> Click the plus icon --> Select "Visualization" tab --> Search for "Squid Event Count" --> Select it.
   The visualization will be added to the bottom of the dashboard.
6. Click the save icon on the top right to save the dashboard.

Creating an Alert Detail Panel 

1. Log into the Metron UI Dashboard: http://METRON_UI_HOST:5000.
2. Select "Discover" Tab --> Select the "squid*" index.
3. Search only for alerts in the Squid index.
   1. Type the following in search:
      "is_alert = true"
   2. Click the search icon
4. Now we only need to select a subset of the fields that we want to display in the detail panel. In the left hand panel under "Available Fields", add the following fields:

full_hostname
ip_src_addr
ip_dst_addr
original_string
method
type

Dashboard with the Two Panels

The following is what the new dashboard will look like with these two new panels. 

Image Added

 Image Removed