THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
The heart of the problem is how one defines a "condition." In Metron, we provide a custom domain specific language named "Stellar" for defining conditions.
The query language supports the following:
- Referencing fields in the enriched JSON
- Simple boolean operations:
and, &¬or, ||
- Determining whether a field exists (via
exists) - The ability to have parenthesis to make order of operations explicit
- A fixed set of functions which take strings and return boolean. Currently:
IN_SUBNET(ip, cidr1, cidr2, ...)IS_EMPTY(str)STARTS_WITH(str, prefix)ENDS_WITH(str, suffix)REGEXP_MATCH(str, pattern)
- A fixed set of string to string transformation functions. Currently:
TO_LOWERTO_UPPERTRIM
The documentation can be found here - https://github.com/apache/incubator-metron/blob/master/metron-platform/metron-common/README.md
Consider, for example, the following JSON message:
...
Now that we have the ability to define conditions, for each sensor we need to associate these conditions to scores. Since this is a per-sensor configuration, this fits nicely within the sensor enrichment configuration held in zookeeper. This configuration fits well within the threatIntel section of the configuration like so:
{...,,"threatIntel" : {...,, "triageConfig" : {"riskLevelRules" : [{"condition1name" : level1"rule1",, "condition2comment" : level2"comment1","rule" : "<...","score" : 5,"reason" : "some reason"},{"name" : "rule2","comment" : "comment2","rule" : "<...","score" : 10,"reason"aggregator" : "MAXsome reason"},...],"aggregator" : "MAX"}}}
riskLevelRules correspond to the set of condition to numeric level mappings that define the threat triage for this particular sensor. aggregator is an aggregation function that takes all non-zero scores representing the matching queries from riskLevelRules and aggregates them into a single score. The current supported aggregation functions are
...
{...,"threatIntel" : {...,, "triageConfig" : {"riskLevelRules" : { [
{"name" : "in_zeus","comment" : "Checks if this domain without subdomains matches against the zeus threat intel list","reason" : "FORMAT('%s exists in the Zeus threat intel list', domain_without_subdomains)","rule" : "exists(threatintels.hbaseThreatIntel.domain_without_subdomains.zeusList)","score" : 5},{"name" : "tld_check","comment" : "Applies a risk score based on the domain TLD","reason" : "FORMAT('%s does not end with com or net', domain_without_subdomains)","rule" : "not: 5 , "not(ENDS_WITH(domain_without_subdomains, '.com') or ENDS_WITH(domain_without_subdomains, '.net'))","score" : 10},"aggregator" : "MAX" }],"aggregator" : "MAX"}}}
In order to apply this triage configuration, we must modify the configuration for the squid sensor in the enrichment topology. To do this, we should modify $METRON${METRON_HOME}/config/zookeeper/sensors/squid.json on node1. However, since the configuration in zookeeper may have be out of sync with the configuration on disk, we must make sure they are in sync by executing the following command:
$METRON_HOME/bin/zk_load_configs.sh
-m
PULLnode1:2181
-z
$ZOOKEEPER -f
-o
$METRON_HOME/config/zookeeper
We should ensure that the configuration for squid exists by checking out
cat $METRON_HOME/config/zookeeper/enrichments/squid.json
Now we can edit the configuration. In $METRON_HOME/config/zookeeper/enrichments/squid.json edit the section titled riskLevelRules and add the two rules above to the map:
"exists(threatintels.hbaseThreatIntel.domain_without_subdomains.zeusList)" : 5"not(ENDS_WITH(domain_without_subdomains, '.com') or ENDS_WITH(domain_without_subdomains, '.net'))" : 10
array. Also, ensure that the aggregator field indicates MAX.
After modifying the configuration, we can push the configuration back to zookeeper and have the enrichment topology pick it up with live data via
...
tail /var/log/squid/access.log | /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list node1:6667 $BROKERLIST --topic squid
Now, if we check the squid index using the elasticsearch head plugin, we can see the threats triage as we would expect:
...
For URL's from cnn.com, we expect to see no threat alert, so no triage level is set. Notice . Run cnn.com with the Squid client and pipe it into Kafka
squidclient http://www.cnn.com
tail /var/log/squid/access.log -n 1 | /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list $BROKERLIST --topic squid
Notice the lack of a threat.triage.level field.
{
"action": "TCP_MISS",
"adapter.simplehbaseadapter.begin.ts": "1492109939268",
"adapter.simplehbaseadapter.end.ts": "1492109939280",
"adapter.threatinteladapter.begin.ts": "1492109939285",
"adapter.threatinteladapter.end.ts": "1492109939289",
"bytes": 128477,
"code": 200,
"domain_without_subdomains": "cnn.com",
"elapsed": 25,
"enrichmentjoinbolt.joiner.ts": "1492109939282",
"enrichments.hbaseEnrichment.domain_without_subdomains.whois.domain": "cnn.com",
"enrichments.hbaseEnrichment.domain_without_subdomains.whois.domain_created_timestamp": "748695600000",
"enrichments.hbaseEnrichment.domain_without_subdomains.whois.home_country": "US",
"enrichments.hbaseEnrichment.domain_without_subdomains.whois.owner": "Turner Broadcasting System, Inc.",
"enrichments.hbaseEnrichment.domain_without_subdomains.whois.registrar": "Domain Name Manager",
"enrichmentsplitterbolt.splitter.begin.ts": "1492109939265",
"enrichmentsplitterbolt.splitter.end.ts": "1492109939265",
"full_hostname": "www.cnn.com",
"guid": "bdf0d0de-3f6d-4479-848b-1c56e06050de",
"ip_dst_addr": "151.101.41.67",
"ip_src_addr": "::1",
"method": "GET",
"original_string": "1492109922.444 25 ::1 TCP_MISS/200 128477 GET http://www.cnn.com/ - DIRECT/151.101.41.67 text/html",
"source.type": "squid",
"threatinteljoinbolt.joiner.ts": "1492109939291",
"threatintelsplitterbolt.splitter.begin.ts": "1492109939283",
"threatintelsplitterbolt.splitter.end.ts": "1492109939283",
"timestamp": 1492109922444,
"url": "http://www.cnn.com/"
}
Threat Data from alamman.com has a triage level of 5
Because alammanwebtahmin.com is com is a malicious host from the zeusList threat intel feed but is a .com address, it's assigned threat.triage.level of 5.
{
"action": "TCP_MISS",
"adapter.simplehbaseadapter.begin.ts": "1492109261268",
"adapter.simplehbaseadapter.end.ts": "1492109261273",
"adapter.threatinteladapter.begin.ts": "1492109261279",
"adapter.threatinteladapter.end.ts": "1492109261287",
"bytes": 69540,
"code": 200,
"domain_without_subdomains": "webtahmin.com",
"elapsed": 4288,
"enrichmentjoinbolt.joiner.ts": "1492109261274",
"enrichmentsplitterbolt.splitter.begin.ts": "1492109261266",
"enrichmentsplitterbolt.splitter.end.ts": "1492109261266",
"full_hostname": "webtahmin.com",
"guid": "cfb72fe1-376a-4850-b2b2-acd36a1f7bf7",
"ip_dst_addr": "185.59.28.14",
"ip_src_addr": "::1",
"is_alert": "true",
"method": "GET",
"original_string": "1492109249.738 4288 ::1 TCP_MISS/200 69540 GET http://webtahmin.com/ - DIRECT/185.59.28.14 text/html",
"source.type": "squid",
"threat.triage.rules.0.comment": "Checks if this domain without subdomains matches against the zeus threat intel list",
"threat.triage.rules.0.name": "in_zeus",
"threat.triage.rules.0.reason": "webtahmin.com exists in the Zeus threat intel list",
"threat.triage.rules.0.score": 5,
"threat.triage.score": 5.0,
"threatinteljoinbolt.joiner.ts": "1492109261293",
"threatintels.hbaseThreatIntel.domain_without_subdomains.zeusList": "alert",
"threatintelsplitterbolt.splitter.begin.ts": "1492109261276",
"threatintelsplitterbolt.splitter.end.ts": "1492109261276",
"timestamp": 1492109249738,
"url": "http://webtahmin.com/"
}
Threat Data from atmape.ru has a triage level of 10
Because atmape.ru is both a malicious host from the zeusList threat intel feed as well as a non .com and non .net address, it's assigned threat.triage.level of 10.
{
"action": "TCP_MEM_HIT",
"adapter.simplehbaseadapter.begin.ts": "1492108679325",
"adapter.simplehbaseadapter.end.ts": "1492108679329",
"adapter.threatinteladapter.begin.ts": "1492108679336",
"adapter.threatinteladapter.end.ts": "1492108679347",
"bytes": 3654,
"code": 200,
"domain_without_subdomains": "atmape.ru",
"elapsed": 0,
"enrichmentjoinbolt.joiner.ts": "1492108679331",
"enrichmentsplitterbolt.splitter.begin.ts": "1492108679324",
"enrichmentsplitterbolt.splitter.end.ts": "1492108679324",
"full_hostname": "www.atmape.ru",
"guid": "524aea4d-f04e-42f4-b5cf-33c8a8e1ae3b",
"ip_src_addr": "::1",
"is_alert": "true",
"method": "GET",
"original_string": "1492108654.717 0 ::1 TCP_MEM_HIT/200 3654 GET http://www.atmape.ru/ - NONE/- text/html",
"source.type": "squid",
"threat.triage.rules.0.comment": "Checks if this domain without subdomains matches against the zeus threat intel list",
"threat.triage.rules.0.name": "in_zeus",
"threat.triage.rules.0.reason": "atmape.ru exists in the Zeus threat intel list",
"threat.triage.rules.0.score": 5,
"threat.triage.rules.1.comment": "Applies a risk score based on the domain TLD",
"threat.triage.rules.1.name": "tld_check",
"threat.triage.rules.1.reason": "atmape.ru does not end with com or net",
"threat.triage.rules.1.score": 10,
"threat.triage.score": 10.0,
"threatinteljoinbolt.joiner.ts": "1492108679349",
"threatintels.hbaseThreatIntel.domain_without_subdomains.zeusList": "alert",
"threatintelsplitterbolt.splitter.begin.ts": "1492108679334",
"threatintelsplitterbolt.splitter.end.ts": "1492108679334",
"timestamp": 1492108654717,
"url": "http://www.atmape.ru/"
}
s{Wwwwww
We{hreatintels.hbaseT{hreatIntel.url.zeusList