HackIllinois2016+Eagle

Eagle project was invited to HackIllinois event https://www.hackillinois.org/

One idea is to develop security event correlation platform for user to easily onboard new metric and model metric correlation.

In order for students to focus on correlation platform design, we won't use Eagle existing HBase storage, Rest API, DSL, and dynamic policy framework. Instead, we create this prototype based on mysql, and students have to write new API and build storm topology from scratch.

The basic requirements are:

1. Design tables: metricSchemaDef, metricGroupDef, policyDef
2. Storm topology: one spout + N correlation bolts
3. Spout
   1. reference KafkaSpout, but may need some fundamental changes if we want support multiple topic
   2. maintain a route table which maps metricGroup to correlation bolt ID. This table should be identical across all spout instances
   3. read metricGroupDef and metricSchemaDef in background and spawn new thread to read new metric
4. Correlation bolt
   1. read policyDef

Engineering Design

Implementations:

1. Example code is under https://github.com/yonzhang/incubator-eagle/tree/hackillinois
   1. eagle-examples/eagle-correlation-engine   topology example to wrap multiple KafkaSpout into one spout
   2. eagle-examples/eagle-correlation-service   service example to provide API to read/write metadata, for example metric, metric group etc.