Page History

...

Now that the Grok pattern is staged in HDFS we need to define Storm Flux configuration a parser configuration for the Metron Parsing Topology.  The configs are staged under 

/usr/metron/0.1BETA/config/topologies/ and each parsing topology has it's own set of configs.  Each directory for a topology has a remote.yaml which is designed to be run on AWS and local/test.yaml designed to run locally on a single-node VM.  At the moment of publishing this blog entry the following configs are available:

/usr/metron/0.1BETA/flux/test.yaml

/usr/metron/0.1BETA/flux/remote.yaml

/usr/metron/0.1BETA/flux/sourcefire/test.yaml

/usr/metron/0.1BETA/flux/sourcefire/remote.yaml

/usr/metron/0.1BETA/flux/asa/test.yaml

/usr/metron/0.1BETA/flux/asa/remote.yaml

/usr/metron/0.1BETA/flux/fireeye/test.yaml

/usr/metron/0.1BETA/flux/fireeye/remote.yaml

/usr/metron/0.1BETA/flux/bro/test.yaml

/usr/metron/0.1BETA/flux/bro/remote.yaml

/usr/metron/0.1BETA/flux/ise/test.yaml

/usr/metron/0.1BETA/flux/ise/remote.yaml

/usr/metron/0.1BETA/flux/paloalto/test.yaml

/usr/metron/0.1BETA/flux/paloalto/remote.yaml

/usr/metron/0.1BETA/flux/lancope/test.yaml

/usr/metron/0.1BETA/flux/lancope/remote.yaml

/usr/metron/0.1BETA/flux/pcap/test.yaml

/usr/metron/0.1BETA/flux/pcap/remote.yaml

/usr/metron/0.1BETA/flux/enrichment/test.yaml

/usr/metron/0.1BETA/flux/enrichment/remote.yaml

/usr/metron/0.1BETA/flux/snort/test.yaml

/usr/metron/0.1BETA/flux/snort/remote.yaml

Since we are going to be running locally on a VM we need to define a test.yaml for Squid.  The easiest way to do this is to copy one of the existing Grok-based configs (YAF) and tailor it for Squid.  

mkdir /usr/metron/0.1BETA/flux/squid

cp /usr/metron/0.1BETA/flux/yaf/remote.yaml /usr/metron/0.1BETA/flux/squid/remote.yaml

vi /usr/metron/0.1BETA/flux/squid/remote.yaml

And edit your config to look like this (changes highlighted in red):

configurations are kept in Zookeeper so the sensor configuration must be uploaded there after it has been created.  A Grok parser configuration follows this format:

{
  "parserClassName": "org.apache.metron.parsers.GrokParser",
  "sensorTopic": "sensor name",
  "parserConfig": {
    "grokPath": "grok pattern",
    "patternLabel": "grok label",
    ... other optional fields
  }
}

Create a Squid Grok parser configuration file at /usr/metron/0.1BETA/config/zookeeper/parsers/squid.json with the following contents:

name: "squid"

config:

    topology.workers: 1

 

 

components:

    -   id: "parser"

        className: "org.apache.metron.parsers.GrokParser"

        constructorArgs:

            - "/apps/metron/patterns/squid"

            - "SQUID_DELIMITED"

        configMethods:

            -   name: "withTimestampField"

                args:

                    - "timestamp"

    -   id: "writer"

        className: "org.apache.metron.parsers.writer.KafkaWriter"

        constructorArgs:

            - "${kafka.broker}"

    -   id: "zkHosts"

        className: "storm.kafka.ZkHosts"

        constructorArgs:

            - "${kafka.zk}"

    -   id: "kafkaConfig"

        className: "storm.kafka.SpoutConfig"

        constructorArgs:

            # zookeeper hosts

            - ref: "zkHosts"

            # topic name

            - "squid"

            # zk root

            - ""

            # id

            - "squid"

        properties:

            -   name: "ignoreZkOffsets"

                value: true

            -   name: "startOffsetTime"

                value: -1

            -   name: "socketTimeoutMs"

                value: 1000000

 

spouts:

    -   id: "kafkaSpout"

        className: "storm.kafka.KafkaSpout"

        constructorArgs:

            - ref: "kafkaConfig"

 

bolts:

    -   id: "parserBolt"

        className:

{
  "parserClassName": 

"org.apache.metron.parsers.

bolt.ParserBolt"

        constructorArgs:

            - "${kafka.zk}"

            - "squid"

            - ref: "parser"

            - ref: "writer"

 

streams:

    -   name: "spout -> bolt"

        from: "kafkaSpout"

        to: "parserBolt"

        grouping:

            type: SHUFFLE

Start the new squid parser topology:

GrokParser",
  "sensorTopic": "squid",
  "parserConfig": {
    "grokPath": "/apps/metron/patterns/squid",
    "patternLabel": "SQUID_DELIMITED",
    "timestampField": "timestamp"
  }
}

A script is provided to upload configurations to Zookeeper.  Upload the new parser config to Zookeeper:

storm jar /usr/metron/0.1BETA/lib/metron-parsers-0.1BETA.jar org.apache.storm.flux.Flux --filter bin/zk_load_configs.sh --mode PUSH -i /usr/metron/0.1BETA/config/elasticsearch.properties --remote zookeeper -z node1:2181 

Start the new squid parser topology:

/usr/metron/0.1BETA/flux/squid/remote.yamlbin/start_parser_topology.sh -k node1:6667 -z node1:2181 -s squid

Navigate to the squid parser topology in the Storm UI at http://node1:8744/index.html and verify the topology is up with no errors:

...