You are viewing an old version of this page. View the current version.

Compare with Current View Page History

Version 1 Next »

When developing new parsers (or making other changes to the parser or enrichment topologies), it's important to be able to troubleshoot problems when things don't act as expected.  This blog will cover techniques for troubleshooting the various Metron components.  It is assumed that you have followed the last few sections on adding the new data source.  Before starting, make sure your IDE is setup by following the instructions here.

Troubleshooting Parser Topologies

The integration testing framework can be a very effective way to troubleshoot topologies because they not only allow you to test parser logic (which is hopefully being done in an accompanying parser unit test) but also the related flux and property files.  Adding an integration test is highly recommended regardless of whether detailed troubleshooting is needed or not.  The following steps describe the process for setting up and stepping through an integration test.  The Squid parser that was created in Part 1 of the Metron Tutorial Fundamentals blog series will be used an example.

Create a test flux file

A test flux file very similar to the flux file used for running topologies on a cluster needs to be created.  Copy the flux file create for this parser topology at /incubator-metron/metron-platform/metron-parsers/src/main/flux/squid/remote.yaml into the same directory only renamed to test.yaml.  Adjust the test.yaml to use a local path for the grok file and read from the beginning of the Kafka topic:

Create some sample data

The first step is to create some sample data.  Initially this can be just a couple lines, similar to what was used to write a unit test.  Create a file called SquidExampleOutput in the sample input data directory (/incubator-metron/metron-platform/metron-integration-test/src/main/resources/sample/data/SampleInput) with the following lines:

1461576382.642    161 127.0.0.1 TCP_MISS/200 103701 GET http://www.cnn.com/ - DIRECT/199.27.79.73 text/html
1461576442.228    159 127.0.0.1 TCP_MISS/200 137183 GET http://www.nba.com/ - DIRECT/66.210.41.9 text/html

The integration test will test the accuracy of the parser topology by comparing parsed data against a set of expected parsed data.  Create a file called SquidExampleParsed in the sample parsed directory (/incubator-metron/metron-platform/metron-integration-test/src/main/resources/sample/data/SampleParsed) with the following lines:

{"elapsed":161,"code":200,"ip_dst_addr":"199.27.79.73","original_string":"1461576382.642    161 127.0.0.1 TCP_MISS\/200 103701 GET http:\/\/www.cnn.com\/ - DIRECT\/199.27.79.73 text\/html","method":"GET","bytes":103701,"action":"TCP_MISS","ip_src_addr":"127.0.0.1","url":"cnn.com","timestamp":1461576382642,"source.type":"squid"}
{"elapsed":159,"code":200,"ip_dst_addr":"66.210.41.9","original_string":"1461576442.228    159 127.0.0.1 TCP_MISS\/200 137183 GET http:\/\/www.nba.com\/ - DIRECT\/66.210.41.9 text\/html","method":"GET","bytes":137183,"action":"TCP_MISS","ip_src_addr":"127.0.0.1","url":"nba.com","timestamp":1461576442228,"source.type":"squid"}

Create an integration test

The infrastructure for running an integration test can easily be leveraged by extending the base parser integration test.  Create a java class called SquidIntegrationTest in /incubator-metron/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/integration that extends ParserIntegrationTest.  The ParserIntegrationTest is an abstract class that requires a couple of methods to be implemented.  These methods should be fairly intuitive (SnortIntegrationTest and YafIntegrationTest can be referenced as examples) and include providing the location of assets created earlier in this tutorial (test flux file, sample input data path, sample parsed data path) and the type of sensor:

In out example the getFluxTopicProperty method can be empty because the Kafka topics are hardcoded in the flux files.  Now you are setup to run an integration test for the Squid parser.

Set some breakpoints

Adding break points in the ParserBolt.prepare and ParserBolt.execute methods should provide a good starting point to troubleshooting parser topologies:

Run the test

Now run the integration test in Debug mode by either creating a Run/Debug Configuration:

 

Or simply right-clicking inside the integration test and selecting "Debug 'SquidIntegrationTest'":

You should now be able to step through the parser topology and see exactly what's going on:

  • No labels