...
cat /var/log/squid/access.log | ${HDP_HOME}/kafka-broker/bin/kafka-console-producer.sh --broker-list $BROKERLIST --topic squid
${HDP_HOME}/kafka-broker/bin/kafka-console-consumer.sh --zookeeper $ZOOKEEPER bootstrap-server $BROKERLIST --topic squid --from-beginning
...
curl -XPUT 'http://node1:9200/_template/squid_index' -d '{"template": "squid_index*","mappings": {"squid_doc": {"dynamic_templates": [{"geo_location_point": {"match": "enrichments:geo:*:location_point","match_mapping_type": "*","mapping": {"type": "geo_point"}}},{"geo_country": {"match": "enrichments:geo:*:country","match_mapping_type": "*","mapping": {"type": "keyword"}}},{"geo_city": {"match": "enrichments:geo:*:city","match_mapping_type": "*","mapping": {"type": "keyword"}}},{"geo_location_id": {"match": "enrichments:geo:*:locID","match_mapping_type": "*","mapping": {"type": "keyword"}}},{"geo_dma_code": {"match": "enrichments:geo:*:dmaCode","match_mapping_type": "*","mapping": {"type": "keyword"}}},{"geo_postal_code": {"match": "enrichments:geo:*:postalCode","match_mapping_type": "*","mapping": {"type": "keyword"}}},{"geo_latitude": {"match": "enrichments:geo:*:latitude","match_mapping_type": "*","mapping": {"type": "float"}}},{"geo_longitude": {"match": "enrichments:geo:*:longitude","match_mapping_type": "*","mapping": {"type": "float"}}},{"timestamps": {"match": "*:ts","match_mapping_type": "*","mapping": {"type": "date","format": "epoch_millis"}}},{"threat_triage_score": {"mapping": {"type": "float"},"match": "threat:triage:*score","match_mapping_type": "*"}},{"threat_triage_reason": {"mapping": {"type": "text","fielddata": "true"},"match": "threat:triage:rules:*:reason","match_mapping_type": "*"}},{"threat_triage_name": {"mapping": {"type": "text","fielddata": "true"},"match": "threat:triage:rules:*:name","match_mapping_type": "*"}}],"properties": {"timestamp": {"type": "date","format": "epoch_millis"},"source:type": {"type": "keyword"},"ip_dst_addr": {"type": "ip"},"ip_dst_port": {"type": "integer"},"ip_src_addr": {"type": "ip"},"ip_src_port": {"type": "integer"},"alert": {"type": "nested"},"metron_alert" : {
"type" : "nested"
},"guid": {
"type": "keyword"}}}}}'# Verify the template installs as expectedcurl -XGET 'http://node1:9200/_template/squid_index?pretty'
...
- Sets up default mappings for metron-specific types, e.g. timestamps.
- Sets up types for properties that will come from the parsed data, e.g. ip_src_addr.
If you're using the Full dev environment, you might want to stop some of the other parsers to free up resources.
for parser in bro__snort__yaf profiler pcap batch_indexing; do storm kill parser; done
Now start the new squid parser topology:
...