Page History

...

Our example sensor will be a Squid Proxy.  Squid is a caching proxy for the Web supporting HTTP, HTTPS, FTP, and more.  Squid logs are simple to explain and easy to parse and the velocity of traffic coming from Squid is representative of a a typical network-based sensor.  Hence, we feel it's a good telemetry to use for this tutorial.

Step 1: Aquire Metron Code and Development Environment 

 There are two ways to acquire Metron code for this code lab.  One is to download it from the USB stick administered for this exercise.  Two it would automatically be imported by running the code lab platform vagrant scripts 

...

By running the following script if you have the local copy of the code lab image from the USB stick it will use the USB version, but otherwise will get the image from Vagrant Atlas.  Beware the image is large so it will take a little while to download it.   

Step 2: Build the Metron code (Optional)

If you are not running Metron from the USB stick you need to download and build the code.   Please see here for full Metron installation and validation instructions.  Verify that the project has been built before creating the VM.  First lets get Metron from Apache.

...

Now we have downloaded and built metron it's on to the next step.  Next we need to make a decision about the Metron environment and which parts of Metron we would like to build.  If you are running from the USB stick the code is already pre-built. 

Step 3 : Installing a sample sensor

Log into the sensors node and install the squid sensor.  If you are on the QuickDev platform your VM will be called node1.  If you are on AWS environment your sensor node will be tagged with the [sensors] tag.  You can look through the AWS console to find which node in your cluster has this tag.  

...

Now that we have the sensor set up and generating logs we need to figure out how to pipe these logs to a Kafka topic.  To do so the first thing we need to do is setup a new Kafka topic for Squid.

Step 4 : Define Environment Variables 

If you are using the quick-dev image your links are:

...

export METRON_VERSION="0.2.0BETA"

Step 5 : Create Kafka topics and ingest sample data 

/usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper $ZOOKEEPER:2181 --create --topic squid --partitions 1 --replication-factor 1

/usr/hdp/current/kafka-broker/bin//kafka-topics.sh --zookeeper $ZOOKEEPER:2181 --list

...