Page History

...

The integration testing framework can be a very effective way to troubleshoot topologies because they not only allow you to test parser logic (which is hopefully being done in an accompanying parser unit test) but also the related flux and property configuration files.  Adding an integration test is highly recommended regardless of whether detailed troubleshooting is needed or not.  The following steps describe the process for setting up and stepping through an integration test.  The Squid parser that was created in Part 1 of the Metron Tutorial Fundamentals blog series will be used an example.

Create

...

A test flux file very similar to the flux file used for running topologies on a cluster needs to be created.  Copy the flux file create for this parser topology at /incubator-metron/metron-platform/metron-parsers/src/main/flux/squid/remote.yaml into the same directory only renamed to test.yaml.  Adjust the test.yaml to use a local path for the grok file and read from the beginning of the Kafka topic:

Image Removed

Create some sample data

The first step is to create some sample data.  Initially this can be just a couple lines, similar to what was used to write a unit test.  Parser integration tests can automatically locate sample data as long as the paths follows these patterns:

/incubator-metron/metron-platform/metron-integration-test/src/main/sample/data/<sensor type>/raw (for raw data)

/incubator-metron/metron-platform/metron-integration-test/src/main/sample/data/<sensor type>/parsed (for parsed data)

Create  Create a file called SquidExampleOutput in the sample input data directory ( /incubator-metron/metron-platform/metron-integration-test/src/main/resources/sample/data/SampleInput) with squid/raw with the following lines:

1461576382.642    161 127.0.0.1 TCP_MISS/200 103701 GET http://www.cnn.com/ - DIRECT/199.27.79.73 text/html
1461576442.228    159 127.0.0.1 TCP_MISS/200 137183 GET http://www.nba.com/ - DIRECT/66.210.41.9 text/html

Image RemovedImage Added

The parser integration test will test the accuracy of the parser topology by comparing parsed raw data against a set of expected parsed data.  Create a file called SquidExampleParsed in the sample parsed directory ( /incubator-metron/metron-platform/metron-integration-test/src/main/resources/sample/data/SampleParsed) with squid/parsed with the following lines:

{"elapsed":161,"code":200,"ip_dst_addr":"199.27.79.73","original_string":"1461576382.642    161 127.0.0.1 TCP_MISS\/200 103701 GET http:\/\/www.cnn.com\/ - DIRECT\/199.27.79.73 text\/html","method":"GET","bytes":103701,"action":"TCP_MISS","ip_src_addr":"127.0.0.1","url":"cnn.com","timestamp":1461576382642,"source.type":"squid"}
{"elapsed":159,"code":200,"ip_dst_addr":"66.210.41.9","original_string":"1461576442.228    159 127.0.0.1 TCP_MISS\/200 137183 GET http:\/\/www.nba.com\/ - DIRECT\/66.210.41.9 text\/html","method":"GET","bytes":137183,"action":"TCP_MISS","ip_src_addr":"127.0.0.1","url":"nba.com","timestamp":1461576442228,"source.type":"squid"}

Image RemovedImage Added

Create an integration test

The infrastructure for running an integration test can easily be leveraged by extending the base parser integration test.  Create a java class called SquidIntegrationTest in /incubator-metron/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/integration that extends ParserIntegrationTest.  The ParserIntegrationTest is an abstract class that requires a couple of methods to be implemented.  These methods should be fairly intuitive (SnortIntegrationTest and YafIntegrationTest can be referenced as examples) and include providing the location of assets created earlier in this tutorial (test flux file, sample input data path, sample parsed data path) and the type of sensor:

Image Removed

sensor type and validations that should be performed:

Image Added

We will use In out example the getFluxTopicProperty method can be empty because the Kafka topics are hardcoded in the flux files .  Now you are setup to run an integration test for the Squid parser.

...

Adding break points in the ParserBolt.prepare and ParserBolt.execute methods should provide a good starting point to troubleshooting parser topologies:

Image Modified

Run the test

Now run the integration test in Debug mode by either creating a Run/Debug Configuration:

Image Modified

Or simply right-clicking inside the integration test and selecting "Debug 'SquidIntegrationTest'":

Image Modified

You should now be able to step through the parser topology and see exactly what's going on:

Image Modified