Page History

...

Reference ambari url 

0.xBETA

In this blog post we will walk through what it takes to setup a new telemetry source in Metron.  For this example we will setup a new sensor, capture the sensor logs, pipe the logs to Kafka, pick up the logs with a Metron parsing topology, parse them, and run them through the Metron stream processing pipeline.  

Our example sensor will be a a Squid Proxy.  Squid is a caching proxy for the Web supporting HTTP, HTTPS, FTP, and more.  Squid logs are simple to explain and easy to parse and the velocity of traffic coming from Squid is representative of a a typical network-based sensor.  Hence, we feel it's a good telemetry to use for this tutorial. 

Step 1: Build the Metron code 

Prior to going through this tutorial make sure you have Metron properly installedbuilt and tested.  Please  Please see here for  for full Metron installation and validation instructions.  Verify that the project has been built before creating the VM:

cd metron-platform

mvn clean package

.  First lets get Metron from Apache.

git clone https://git-wip-us.apache.org/repos/asf/incubator-metron.git

git tag -l

Now you will see a list of Metron releases.  You will see major releases, minor releases, and release candidaes.  Refer to the Metron website with regards to which is the current stable release recommended for downloading.  Once you select the Metron release run the following comand to download it:

cd incubator-metron

git checkout tags/[MetronReleaseVersion]

Now that we have downloaded Metron we need to build it.  For the purposes of this exercise we will build without running through Metron's unit and integration test suites.  To do so run the following command:

mvn clean package

Now we have downloaded and built metron it's on to the next step.  Next we need to make a decision about the Metron environment and which parts of Metron we would like to build.  For the purpose of this exercise we will assume that we are building full metron on the QuickDev image or Amazon AWS.  We will provide instructions for both.  

Step 2a : Setup Setup the QuickDev Image

If you want to take Metron for a spin local on your laptop you need to setup the QuickDev environment.  The QuickDev environment is primarily intended for developers and people who want to spin up Metron quickly without incurring costs on AWS.  But fair warning, this environment is not meant for production and is not performant.  It is is intended merely for demonstration and development.  To spin up QuickDev perform the following steps:

 We will be using a single VM setup for this exercise.  To setup the VM do the following steps:

vagrant plugin install vagrant-hostmanager

cd metron-deployment/vagrant/quick-dev-platform

./launch_dev_image.sh

vagrant ssh

After executing the above commands a Metron VM will be built (called node1) and you will be logged in as user vagrant.  There will be 4 topologies running but one must be stopped because the VM only has 4 Storm worker slots available.  

Step 2b : Setup a Full AWS Metron Environment

The AWS environment is intended to install Metron in the AWS cloud.  By default Metron is installed with a few demo sensors. 

Leave the enrichment topology running and kill the other parser topologies (bro, snort, or yaf) with either the "storm kill" command or with the Storm UI at http://node1:8744/index.html.  Now lets install the Squid sensor.  

...