Eagle project was invited to HackIllinois event https://www.hackillinois.org/

Thanks Tyler Brown, Subasree Venkatsubhramaniyen, Yunsheng Wei to make this Hackathon amazing ...

Rest API : Tyler Browner (University of Missouri)

Spout routing: Yunsheng Wei (Urbana-Champaign of Illinois University)

Bolt dispatching: Subasree Venkatsubhramaniyen (University of Wisconsin-Madison)


One idea is to develop security event correlation platform for user to easily onboard new metric and model metric correlation.

In order for students to focus on correlation platform design, we won't use Eagle existing HBase storage, Rest API, DSL, and dynamic policy framework. Instead, we create this prototype based on mysql, and students have to write new API and build storm topology from scratch.

The basic requirements are:

  1. Design tables: metricSchemaDef, metricGroupDef, policyDef
  2. Storm topology: one spout + N correlation bolts
  3. Spout
    1. reference KafkaSpout, but may need some fundamental changes if we want support multiple topic
    2. maintain a route table which maps metricGroup to correlation bolt ID. This table should be identical across all spout instances
    3. read metricGroupDef and metricSchemaDef in background and spawn new thread to read new metric
  4. Correlation bolt
    1. read policyDef

 

Engineering Design

eagle security event correlation platform

 

Implementations:

  1. Example code is under https://github.com/yonzhang/incubator-eagle/tree/hackillinois
    1. eagle-examples/eagle-correlation-engine   topology example to wrap multiple KafkaSpout into one spout
    2. eagle-examples/eagle-correlation-service   service example to provide API to read/write metadata, for example metric, metric group etc.

 

  • No labels