...
Now that the Grok pattern is staged in HDFS we need to define Storm Flux configuration a parser configuration for the Metron Parsing Topology. The configs are staged under
/usr/metron/0.1BETA/config/topologies/ and each parsing topology has it's own set of configs. Each directory for a topology has a remote.yaml which is designed to be run on AWS and local/test.yaml designed to run locally on a single-node VM. At the moment of publishing this blog entry the following configs are available:
/usr/metron/0.1BETA/flux/test.yaml
/usr/metron/0.1BETA/flux/remote.yaml
/usr/metron/0.1BETA/flux/sourcefire/test.yaml
/usr/metron/0.1BETA/flux/sourcefire/remote.yaml
/usr/metron/0.1BETA/flux/asa/test.yaml
/usr/metron/0.1BETA/flux/asa/remote.yaml
/usr/metron/0.1BETA/flux/fireeye/test.yaml
/usr/metron/0.1BETA/flux/fireeye/remote.yaml
/usr/metron/0.1BETA/flux/bro/test.yaml
/usr/metron/0.1BETA/flux/bro/remote.yaml
/usr/metron/0.1BETA/flux/ise/test.yaml
/usr/metron/0.1BETA/flux/ise/remote.yaml
/usr/metron/0.1BETA/flux/paloalto/test.yaml
/usr/metron/0.1BETA/flux/paloalto/remote.yaml
/usr/metron/0.1BETA/flux/lancope/test.yaml
/usr/metron/0.1BETA/flux/lancope/remote.yaml
/usr/metron/0.1BETA/flux/pcap/test.yaml
/usr/metron/0.1BETA/flux/pcap/remote.yaml
/usr/metron/0.1BETA/flux/enrichment/test.yaml
/usr/metron/0.1BETA/flux/enrichment/remote.yaml
/usr/metron/0.1BETA/flux/snort/test.yaml
/usr/metron/0.1BETA/flux/snort/remote.yaml
Since we are going to be running locally on a VM we need to define a test.yaml for Squid. The easiest way to do this is to copy one of the existing Grok-based configs (YAF) and tailor it for Squid.
mkdir /usr/metron/0.1BETA/flux/squid
cp /usr/metron/0.1BETA/flux/yaf/remote.yaml /usr/metron/0.1BETA/flux/squid/remote.yaml
vi /usr/metron/0.1BETA/flux/squid/remote.yaml
And edit your config to look like this (changes highlighted in red):
configurations are kept in Zookeeper so the sensor configuration must be uploaded there after it has been created. A Grok parser configuration follows this format:
{
"parserClassName": "org.apache.metron.parsers.GrokParser",
"sensorTopic": "sensor name",
"parserConfig": {
"grokPath": "grok pattern",
"patternLabel": "grok label",
... other optional fields
}
}
Create a Squid Grok parser configuration file at /usr/metron/0.1BETA/config/zookeeper/parsers/squid.json with the following contents:
name: "squid"
config:
topology.workers: 1
components:
- id: "parser"
className: "org.apache.metron.parsers.GrokParser"
constructorArgs:
- "/apps/metron/patterns/squid"
- "SQUID_DELIMITED"
configMethods:
- name: "withTimestampField"
args:
- "timestamp"
- id: "writer"
className: "org.apache.metron.parsers.writer.KafkaWriter"
constructorArgs:
- "${kafka.broker}"
- id: "zkHosts"
className: "storm.kafka.ZkHosts"
constructorArgs:
- "${kafka.zk}"
- id: "kafkaConfig"
className: "storm.kafka.SpoutConfig"
constructorArgs:
# zookeeper hosts
- ref: "zkHosts"
# topic name
- "squid"
# zk root
- ""
# id
- "squid"
properties:
- name: "ignoreZkOffsets"
value: true
- name: "startOffsetTime"
value: -1
- name: "socketTimeoutMs"
value: 1000000
spouts:
- id: "kafkaSpout"
className: "storm.kafka.KafkaSpout"
constructorArgs:
- ref: "kafkaConfig"
bolts:
- id: "parserBolt"
className:{
"parserClassName":"org.apache.metron.parsers.
bolt.ParserBolt"constructorArgs:
- "${kafka.zk}"
- "squid"
- ref: "parser"
- ref: "writer"
streams:
- name: "spout -> bolt"
from: "kafkaSpout"
to: "parserBolt"
grouping:
type: SHUFFLE
Start the new squid parser topology:
GrokParser",
"sensorTopic": "squid",
"parserConfig": {
"grokPath": "/apps/metron/patterns/squid",
"patternLabel": "SQUID_DELIMITED",
"timestampField": "timestamp"
}
}
A script is provided to upload configurations to Zookeeper. Upload the new parser config to Zookeeper:
storm jar /usr/metron/0.1BETA/lib/metron-parsers-0.1BETA.jar org.apache.storm.flux.Flux --filter bin/zk_load_configs.sh --mode PUSH -i /usr/metron/0.1BETA/config/elasticsearch.properties --remote zookeeper -z node1:2181
Start the new squid parser topology:
/usr/metron/0.1BETA/flux/squid/remote.yamlbin/start_parser_topology.sh -k node1:6667 -z node1:2181 -s squid
Navigate to the squid parser topology in the Storm UI at http://node1:8744/index.html and verify the topology is up with no errors:
...