THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
{
"es.clustername": "metron",
"es.ip": "node1",
"es.port": "9300",
"es.date.format": "yyyy.MM.dd.HH",
"fieldValidations" : [
{
"input" : [ "ip_src_addr", "ip_dst_addr" ],
"validation" : "IP",
"config" : {
"type" : "IPV4"
}
}
]
}
Start the new squid parser topology:Now push the global config
/usr/metron/$METRON_VERSION/bin/startzk_parserload_topologyconfigs.sh -k $BROKERLIST:6667 i /usr/metron/$METRON_VERSION/config/zookeeper -m PUSH -z $ZOOKEEPER:2181 -s squid
Navigate to the squid parser topology in the Storm UI at http://node1:8744/index.html and verify the topology is up with no errors:
CREATE ES template before deployment
Now that we have a new running squid parser topology, generate some data to parse by running this command several times:
tail /var/log/squid/access.log | /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list node1:6667 --topic squid
Refresh the Storm UI and it should report data being parsed:
...
/usr/metron/$METRON_VERSION/bin/zk_load_configs.sh -m DUMP -z $ZOOKEEPER:2181
Start the new squid parser topology:
/usr/metron/$METRON_VERSION/bin/start_parser_topology.sh -k $BROKERLIST:6667 -z $ZOOKEEPER:2181 -s squid
Navigate to the squid parser topology in the Storm UI at http://node1:
...
8744/index.html and verify the topology is up with no errors:
Now that we have a new running squid parser topology, generate some data to parse by running this command several times:
sudo tail /var/log/squid/access.log | /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list $BROKERLIST:6667 --topic squid
Refresh the Storm UI and it should report data being parsed:
Then navigate Elasticsearch at http://node1:9200/_cat/indices?v and verify that a squid index has been created:
health status index ...
health status index pri rep docs.count docs.deleted store.size pri.store.sizeyellow open yaf_index_2016.04.25.15 5 1 5485 0 4mb 4mb yellow open snort_index_2016.04.26.12 5 1 24452 pri rep 0 14.4mb 14.4mb docs.count docs.deleted store.size pri.store.size
yellow open broyaf_index_2016.04.25.1615 5 1 12955485 0 1.9mb 4mb 1.9mb4mb
yellow open squidsnort_index_2016.04.26.1312 5 1 24452 1 0 714.3kb4mb 714.3kb4mb
yellow open yafbro_index_2016.04.25.17 5 1 30750 16 5 1 0 1295 17.4mb 17.4mb
In order to verify that the messages were indexed correctly first install elastic search Head plugin:
/usr/share/elasticsearch/bin/plugin -install mobz/elasticsearch-head/1.x
And navigate to http://node1:9200/_plugin/head/
There you will see parsed message + performance timestamps. We will discuss the performance timestamps in another blog entry.
...
0 1.9mb 1.9mbyellow open squid_index_2016.04.26.13 5 1 1 0 7.3kb 7.3kb yellow open yaf_index_2016.04.25.17 5 1 30750 0 17.4mb 17.4mb
In order to verify that the messages were indexed correctly first install elastic search Head plugin:
/usr/share/elasticsearch/bin
sudo plugin install mobz/elasticsearch-head
And navigate to http://node1:9200/_plugin/head/
There you will see parsed message + performance timestamps. We will discuss the performance timestamps in another blog entry.
Now lets see how we create a Kibana dashboard to visualize data in metron. First click on Visualize, select a squid index, and add the fields you wan to display
By convention the index where the new messages will be indexed is called squid_index_[timestamp] and the document type is squid_doc.
Now that we have the messages parsed and indexed we need to setup a Kibana dashboard. To do so access the dashboard on http://node1:5000/#/dashboard/file/default.json
To create a new ingest histogram we first need to setup a pinned query. Click on the query + button and pin a query for _type:squid_doc. This would look like:
Once the query is pinned it will show up in the pinned queries bar like so:
Once the query is established we can create a histogram panel. In the panel settings point the panel to listed to the Squid Logs pinned query you just created
And make sure that the time field points to the field called "timestamp:
Click OK and you should get a histogram that looks like this:
Now to add a detailed telemetry table create a new table panel, and similarly to the histogram panel point it to the Squid Logs pinned query. As a result the following table will be created:
