Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

  1. Drag a processor to the canvas (do this by the dragging the processor icon which is the first icon on the toolbar).
  2. Search for Select the TailFile type of processor, then select Add. 
  3. Right click on the processor and select Configure to display the Configure Processor dialog box. In the Settings tab change the name to "Ingest Squid Events"
    1. In the Properties tab, configure the following:
  4. Drag Another Processor another processor to the canvas.
  5. Search for PutKafka and Select the PutKafka type of processor then select Add.
  6. Right click on the processor and configureselect Configure. 
  7. In the Settings tab, change names the name to "Stream to Metron” then click the checkbox relationship checkboxes for failure and success for relationship.
  8. Under propertiesIn the Properties tab, set 3 the following three properties:
    1. Known Brokers: $KAFKA_HOST:6667
    2. Topic Name: squid
    3. Client Name: nifi-squid
  9. Create a connection by dragging the arrow from the Ingest Squid Events processor to the Stream to Metron processor.
  10. Press the Shift key and select the entire flow, then Select the entire Flow and click the play button (play buttongreen arrow). you You should see all processors green like the of the processor icons turn into green arrows like below:Image Modified
  11. Generate some data using squidclient (do this for about 20+ sites).
    squidclient http://www.cnn.com
  12. You should see metrics on the processor of data being pushed into Metron.
  13. Look at the Storm UI for the parser topology and you should see tuples coming in
  14. After about 5 minutes, you should see a new Elastic Search index called squid_index* in the Elastic Admin UI.

Verify Events are Indexed

...

  1. ssh into Host $SEARCH_HOST.
  2. Install the head plugin:
    usr/share/elasticsearch/bin/plugin -install mobz/elasticsearch-head/1.x 
  3. Navigate to elastic head UI: http://SEARCH_HOST:9200/_plugin/head/
  4. Click on the Browser tab and select squid doc on the left panel and , then select one of th the sample docs. You should see something like the following:Image Modified