Page History

...

1. You should have completed the instructions in Adding a new Telemetry Data Source
2. Make sure the following variables are configured based on your environment: 

   
   KAFKA_HOST = host where a Kafka broker is installed
   ZOOKEEPER_HOST = host where a Zookeeper server is installed
   PROBE_HOST = Host where your sensor, probes are installed. If don't have any sensors installed, pick the host where a storm supervisor is running
   SQUID_HOST = Host where you want to install SQUID. If you don't care, just install on the PROBE_HOST
   NIFI_HOST = The host where you will install NIFI. You want this this to be same host that you installed Squid.
   HOST_WITH_ENRICHMENT_TAG = This is the host in your inventory hosts file that you put under the group "enrichment"
   SEARCH_HOST = This is the host where you have elastic or solr running. This is the host in your inventory hosts file that you put under the group "search". Pick one of the search hosts
   SEARCH_HOST_PORT = The port of the search host where indexing is configured. (e.g: 9300)
   METRON_UI_HOST = This is the host where your metron ui web application is running. This is the host in your inventory hosts file that you put under the group "web".
   METRON_VERSION = The release of the metron binaries you are working with (e.g: 0.2.0BETA-RC2)

    

Step 1: Create a Mock Enrichment Source

...

Step 3: Run the Enrichment Loader

1. Now that we have the enrichment source and enrichment config defined, we can now run the loader to move the data from the enrichment source to the Metron enrichment Store and store the enrichment config in zookeeper.
   1. /usr/metron/

...

1. 1. $METRON_RELEASE/bin/flatfile_loader.sh

...

1. 1. -n

...

1. 1. enrichment_config.json

...

1. 1. -i

...

1. 1. whois_ref.csv

...

1. 1. -t

...

1. 1. enrichment

...

1. 1. -c

...

1. 1. t

...

1. 1. -e

...

1. 1. extractor_config.json

...

2. After this your enrichment data will be loaded in Hbase and a Zookeeper mapping will be established. The data will be populated into Hbase table called enrichment. To verify that the logs were properly ingested into Hbase run the following command: 

   
   hbase

...

1. shell

...

1. 
   scan

...

1. 'enrichment'

...

2. To check if Zookeeper enrichment tag was properly populated, run the following:

   1. /usr/metron/0.1BETA/bin/zk_load_configs.sh

...

1. 1. -m DUMP -z

...

1. 1. ZOOKEEPER_HOST:2181

2. Generate some data by using the squid client to execute http requests (do this about 20

...

1. time

   1. squidclient

...

1. 1. http://www.cnn.com

View the Enrichment Telemetry Events in Metron UI

In order to demonstrate the enrichment capabilities of Metron you need to drop all existing indexes for Squid where the data was ingested prior to enrichments being enabled. To do so go back to the head plugin and deleted the indexes like so:

Image Removed

Make sure you delete all Squid indexes. Re-ingest the data (see previous blog post) and the messages should be automatically enriched. 

In the Metron-UI, refresh the dashboard and view the data in the Squid Panel in the dashboard:

...