THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
This article will introduce Metron's default dashboard that is built upon Kibana 4. It will cover the elements present in the dashboard and how you can extend the dashboard for your own purposes. This is Part 7 of a multi-part tutorial series covering Apache Metron (incubating).
- Metron Tutorial - Fundamentals Part 1: Creating a New Telemetry [DRAFT]
- Metron Tutorial - Fundamentals Part 2: Creating a New Enrichment [DRAFT]
- Metron Tutorial - Fundamentals Part 4: Pluggable Threat Intelligence
- Metron Tutorial - Fundamentals Part 5: Threat Triage
- Metron Tutorial - Fundamentals Part 6: Streaming Enrichment
...
while sleep 2; do cat links.txt | shuf -n 1 | awk -F, '{print $2}' | xargs -i squidclient -g 4 -v {}; done
...
tail -F /var/log/squid/access.log | /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list [kafka_broker]:[kafka_port]$KAFKA_BROKER_URL --topic squid
4. Ensure that the parser topology for Squid continues to run based on the steps outlined in the previous tutorials.
...
1. Run the following command to create an index template for Squid.
curl -XPOST ec2-52-40-44-64.us-west-2.compute.amazonaws.com:9200$ES_HOST:$ES_PORT/_template/squid_index -d '
{
"template": "squid_index*",
"mappings": {
"bro_doc": {
"_timestamp": {
"enabled": true
},
"properties": {
"timestamp": {
"type": "date",
"format": "epoch_millis"
},
"source:type": {
"type": "string",
"index": "not_analyzed"
},
"action": {
"type": "string",
"index": "not_analyzed"
},
"bytes": {
"type": "integer"
},
"code": {
"type": "string",
"index": "not_analyzed"
},
"domain_without_subdomains": {
"type": "string",
"index": "not_analyzed"
},
"full_hostname": {
"type": "string",
"index": "not_analyzed"
},
"elapsed": {
"type": "integer"
},
"method": {
"type": "string",
"index": "not_analyzed"
},
"ip_dst_addr": {
"type": "string",
"index": "not_analyzed"
}
}
}
}
}'
...
3. An index template will only apply for indices that are created after the template is created. Delete the existing Squid indices so that new ones can be generated with the index template.
curl -XDELETE node1:9200/squid*
4. Wait for the Squid index to be re-created. This may take a minute or two based on how fast the Squid data is being consumed in your environment.
curl -XGET node1:9200/squid*
Configure the Squid Index in Kibana
Now that we have a Squid index with all of the right data types, we need to tell Kibana about this index.
| Info |
|---|
Click on the image above to see each of these steps performed. |
1. Login to your Kibana user interface and then click on 'Settings', then 'Indices'.
...
4. Then click the 'Create' button.
Review the Squid Data
Now that Kibana is aware of the new Squid index, let's take a look at the data.
| Info |
|---|
Click on the image above to see each of these steps performed. |
1. Click on `Discover` and then choose the newly created `squid*` index pattern.
...
3. Clicking on a specific record will show each field available in the data.
Save a Squid Search
Let's create a basic data table so that a user can inspect record-level details for Squid. In Kibana, this is done by creating a 'Saved Search'
| Info |
|---|
Click on the image above to see each of these steps performed. |
1. Click on `Discover` and then choose the newly created `squid*` index pattern.
2. In the 'Fields' panel on the left, choose which fields to include in the saved search. Click the 'Add' button next to each field.
3. Click on the 'Save' icon near the top-right to save the search.
Visualize the Squid Data
After using the `Discover` panel to better understand the Squid data, let's create a few visualizations.
| Info |
|---|
Click on the image above to see each of these steps performed. |
1. Click on 'Visualize' in the top level menu.
...
6. Near the top-right side of the screen click on the 'Save' icon to save the visualization. Name it something appropriate. This will allow us to use the visualization in a dashboard later.
Customize the Dashboard
| Info |
|---|
Click on the image above to see each of these steps performed. |
1. Open the Metron Dashboard by clicking on 'Dashboard' in the top-level menu.
...
4. Scroll to the bottom of the dashboard to find the visualization that was added. From here you can resize and move the visualization as needed.
5. Continue enhancing the dashboard by adding the 'Saved Search' that was previously created.
Summary
At this point you should be comfortable customizing a dashboard as you add new sources of telemetry to Metron. This article introduced Metron's default dashboard that is built upon Kibana 4. It covered the elements present in the dashboard and how you can extend the dashboard for your own purposes.
...