Page History

...

For this example we will be using a Zeus malware tracker list located here: https://zeustracker.abuse.ch/blocklist.php?download=domainblocklist.

1. Log into the $HOST_WITH_ENRICHMENT_TAG as root
2. Lets copy the contents from that link to Copy the data form the above link into a file called domainblocklist.txt on your VM..csv 

   
   curl https://zeustracker.abuse.ch/blocklist.php?download=domainblocklist | grep -v "^#" | grep -v "^$" | grep -v "^https" | awk '{print

   Run the following command to parse the above file to a csv file called domainblocklist.csvcat domainblocklist.txt | grep -v "^#" | grep -v "^$" | grep -v "^https" | awk '{print

   $1",abuse.

   ch”

   ch"}'

   >

   domainblocklist.csv

Step 3: Configure an Extractor Config File

1. Log into the $HOST_WITH_ENRICHMENT_TAG as root
2. Now that we have the "Threat Intel Feed Source" , we need to now configure an extractor config file that describes the the source. Create a file called extractor_config_temp.json and put the following contents in it. 

   
   {

   
   "config"

   :

   {

   
       "columns"

   :

   {

   
           "domain"

   :

   0

   
           ,"source"

   :

   1

   
       }

   
       ,"indicator_column"

   :

   "domain"

   
       ,"type"

   :

   "zeusList"

   
       ,"separator"

   :

   ","

   
     }

   
     ,"extractor"

   :

   "CSV"

   
   }

    

3. Run the following to remove the non-ascii characters we run the following:

   1. iconv -c -f utf-8 -t ascii extractor_config_temp.json -o extractor_config.json

...

We now have to configure what element of a tuple and what threat intel feed to cross-reference with.This configuration will be stored in zookeeper.

The config looks like the following:

...

1. Log into the $HOST_WITH_ENRICHMENT_TAG as root
2. Cut and paste this file into a file called "enrichment_config_temp.json" .  

   
   {
        "zkQuorum" : "$ZOOKEEPER_HOST:2181"
       ,"sensorToFieldList"

...

1. :

...

1. {
        "squid" : {
             "type" : "THREAT_INTEL"
            ,"fieldToEnrichmentTypes"

...

1. :

...

1. {
                  "domain_without_subdomains" : [ "zeusList" ]
             }
        }
      }
   }

    

1. Because copying and pasting from this blog will include some non-ascii invisible characters, to strip them out please run

   iconv

...

1. -c

...

1. -f

...

1. utf-8

...

1. -t

...

1. ascii

...

1. enrichment_config_temp.json

...

1. -o

...

1. enrichment_config.json

...

iconv -c -f utf-8 -t ascii enrichment_config_temp.json -o enrichment_config.json

Step 5: Run the Threat Intel Loader

Now that we have the threat intel source,  threat intel exractor and threat intel mapping config defined, we can now run the loader to move the data from the threat intel source to the Metron threat intel Store and store the enrichment config in zookeeper.

1. Log into the $HOST_WITH_ENRICHMENT_TAG as root
2. Run the loader

   
   /usr/metron/

...

1. $METRON_RELEASE/bin/flatfile_loader.sh

...

1. -n

...

1. enrichment_config.json

...

1. -i

...

1. domainblocklist.csv

...

1. -t

...

1. threatintel

...

1. -c

...

1. t

...

1. -e

...

1. extractor_config.json

...

1.  

2. After this, the threat intel data will be loaded in Hbase and a Zookeeper mapping will be established. The data will be populated into Hbase table called threatintel. To verify that the logs were properly ingested into Hbase run the following command: 

   
       hbase

...

1. shell

...

1. 
       scan

...

1. 'threatintel'

...

1.  

2. You should see the table bulk loaded with data from the CSV file. Now check if Zookeeper enrichment tag was properly populated:  

   
   /usr/metron/

...

1. $METRON_RELEASE/bin/zk_load_configs.sh

...

1. -m DUMP -z

...

1. $ZOOKEEPER_HOST:2181

    

    

    
2. Generate some data by using the squid client to execute http requests (do this about 20 times)

   
   squidclient

...

1. http://www.

...

1. actdhaka.com

...

Step 5: View the Threat Alerts in Metron UI

...