Table of Contents

The Setup..

In previous article of the sereies, Enriching Telemetry Events, we walked through how to enrich a domain element of a given telemetry event with WhoIs data like home country, company associated with domain, etc. In this article, we will enrich with a special type of data called threat intel feeds. When a given telemetry event matches data in a threat Intel feed, an alert is generated.

...

In this article, we will walk you through how to do 4 and 5.

Threat Intel Framework Explained

Metron currently provides an extensible framework to plug in threat intel sources. Each threat intel source has two components: an enrichment data source and and enrichment bolt. The threat intelligence feeds are bulk loaded and streamed into a threat intelligence store similarly to how the enrichment feeds are loaded. The keys are loaded in a key-value format. The key is the indicator and the value is the JSON formatted description of what the indicator is. It is recommended to use a threat feed aggregator such as Soltra to dedup and normalize the feeds via Stix/Taxii. Metron provides an adapter that is able to read Soltra-produced Stix/Taxii feeds and stream them into Hbase, which is the data store of choice to back high speed threat intel lookups of Metron. Metron additionally provides a flat file and Stix bulk loader that can normalize, dedup, and bulk load or stream threat intel data into Hbase even without the use of a threat feed aggregator.

The below diagram illustrates the architecture:

Step 1: Setup and Pre-requisites

1. You should have completed the instructions in Adding a new Telemetry Data Source

2. Make sure the following variables are configured based on your environment: 

   
   KAFKA_HOST = host where a Kafka broker is installed
   ZOOKEEPER_HOST = host where a Zookeeper server is installed
   PROBE_HOST = Host where your sensor, probes are installed. If don't have any sensors installed, pick the host where a storm supervisor is running
   SQUID_HOST = Host where you want to install SQUID. If you don't care, just install on the PROBE_HOST
   NIFI_HOST = The host where you will install NIFI. You want this this to be same host that you installed Squid.
   HOST_WITH_ENRICHMENT_TAG = This is the host in your inventory hosts file that you put under the group "enrichment" 
   SEARCH_HOST = This is the host where you have elastic or solr running. This is the host in your inventory hosts file that you put under the group "search". Pick one of the search hosts
   SEARCH_HOST_PORT = The port of the search host where indexing is configured. (e.g: 9300)
   METRON_UI_HOST = This is the host where your metron ui web application is running. This is the host in your inventory hosts file that you put under the group "web".
   METRON_VERSION = The release of the metron binaries you are working with (e.g: 0.2.0BETA-RC2)

Step 2: Create a Mock Threat Intel Feed Source

Metron is designed to work with Stix/Taxii threat feeds, but can also be bulk loaded with threat data from a CSV file. In this example we will explore the CSV example. The same loader framework that is used for enrichment here is used for threat intelligence. Similarly to enrichments we need to setup a data.csv file, the extractor config JSON and the enrichment config JSON.

...

1. Copy the data form the above link into a file called domainblocklist.txt on your VM.
2. Run the following command to parse the above file to a csv file called domainblocklist.csv

   cat domainblocklist.txt | grep -v "^#" | grep -v "^$" | grep -v "^https" | awk '{print $1",abuse.ch”}' > domainblocklist.csv

Step 3: Configure an Extractor Config File

1. Now that we have the "Threat Intel Feed Source" , we need to now configure an extractor config file that describes the the source. Create a file called extractor_config_temp.json and put the following contents in it. 

   1. {
        "config" : {
          "columns" : {
              "domain" : 0
              ,"source" : 1
          }
          ,"indicator_column" : "domain"
          ,"type" : "zeusList"
          ,"separator" : ","
        }
        ,"extractor" : "CSV"
      }

2. Run the following to remove the non-ascii characters we run the following:

   1. iconv -c -f utf-8 -t ascii extractor_config_temp.json -o extractor_config.json

Step

...

4: Configure Element to Threat Intel Feed Mapping

We now have to configure what element of a tuple and what threat intel feed to cross-reference with.This configuration will be stored in zookeeper.

...

iconv -c -f utf-8 -t ascii enrichment_config_temp.json -o enrichment_config.json

Step

...

5: Run the Threat Intel Loader

Now that we have the threat intel source and threat intel config defined, we can now run the loader to move the data from the threat intel source to the Metron threat intel Store and store the enrichment config in zookeeper.

...

squidclient http://www.alamman.com
squidclient http://www.atmape.ru

Step 5: View the Threat Alerts in Metron UI

When the logs are ingested we get messages that has a hit against threat intel:

...