Fault-tolerance
Preload historical data
Deep storage
Policy engine should support state persist and restore. Siddhi is one major policy engine which Eagle currently supports, and Siddhi supports state persist and store.
public interface Snapshotable { public Object[] currentState(); public void restoreState(Object[] state); public String getElementId(); }
In Eagle AlertExecutor is the processing element which handles policy evaluation, policy lifecycle management, so AlertExecutor is the place where policy state can be persisted and restored.
Current policy state = snapshot of policy state + series of events after snapshot
When to do snapshot
Option 1: each AlertExecutor creates a daemon thread which is dedicated for taking snapshot for all polices being processed by this AlertExecutor
Option 2: use storm tick tuple to notify AlertExecutor do a snapshot
While doing snapshot, the whole AlertExecutor is halted to keep state consistent.
How to do delta events persistence
Option1: write-ahead log and move to centralized storage in the backend
Option2: persist each event into Kafka
Option3: persist each event into eagle
Policy state recovery
Recovery only happens when AlertExecutor is first started or restarted by framework.
Recovery includes restoring snapshot and applying delta events