Page History

...

Now lets setup the following environment variables on node1 to make it easier to navigate and carry over the commands from quick-dev to AWS or bare metal deployment  .

export ZOOKEEPER=node1:2181

export BROKERLIST=node1:6667

export HDP_HOME="/usr/hdp/current"

...

export METRON_HOME="/usr/metron/${METRON_VERSION}/""

Note: You should supply a comma-delimited list of host:port items for the ZOOKEEPER and BROKERLIST variables if you are running in an environment with multiple hosts for Zookeeper and the Kafka brokers.

Step 5 : Create Kafka topics and ingest sample data 

/usr/hdp/current${HDP_HOME}/kafka-broker/bin/kafka-topics.sh --zookeeper $ZOOKEEPER :2181 --create --topic squid --partitions 1 --replication-factor 1

/usr/hdp/current/${HDP_HOME}/kafka-broker/bin/kafka-topics.sh --zookeeper $ZOOKEEPER :2181 --list

The following commands will setup a new Kafka topic for squid.  Now let's test how we can pipe the Squid logs to Kakfka

cat /var/log/squid/access.log | /usr/hdp/current ${HDP_HOME}/kafka-broker/bin/kafka-console-producer.sh --broker-list $BROKERLIST :6667 --topic squid

$HDP${HDP_HOME}/kafka-broker/bin/kafka-console-consumer.sh --zookeeper $ZOOKEEPER :2181 --topic squid --from-beginning

...

Create a Squid Grok parser configuration file at /usr/met ron/$METRON_VERSION${METRON_HOME}/config/zookeeper/parsers/squid.json with the following contents:

...

A script is provided to upload configurations to Zookeeper.  Upload the new parser config to Zookeeper:

/usr/metron/$METRON_VERSION${METRON_HOME}/bin/zk_load_configs.sh --mode PUSH -i /usr/metron /$METRON_VERSION${METRON_HOME}/config/zookeeper -z $ZOOKEEPER:2181 

Another thing we can do is validate our messages.  Lets say we wanted to make sure that source IPs and destination IPs are valid.  The validators are global so we set them up on the global JSON and push them into Zookeeper.  To do so perform the following commands:

vi /usr/metron/$METRON_VERSIONvi ${METRON_HOME}/config/zookeeper/global.json

...

Now push the global config

/usr/metron/$METRON_VERSION${METRON_HOME}/bin/zk_load_configs.sh -i /usr/metron/$METRON_VERSION ${METRON_HOME}/config/zookeeper -m PUSH -z $ZOOKEEPER:2181

${METRON_HOME}/usr/metron/$METRON_VERSION/bin/zk_load_configs.sh -m DUMP -z $ZOOKEEPER:2181

Start the new squid parser topology:

/usr/metron/$METRON_VERSION${METRON_HOME}/bin/start_parser_topology.sh -k $BROKERLIST:6667 -z $ZOOKEEPER:2181 -s squid

...

sudo tail /var/log/squid/access.log | /usr/hdp/current/ ${HDP_HOME}/kafka-broker/bin/kafka-console-producer.sh --broker-list $BROKERLIST :6667 --topic squid

Refresh the Storm UI and it should report data being parsed:

...