THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
Status
Current state: Under Discussion
...
- Stateful operators write the state changes to that log (logging the state), in addition to applying them to the state tables in RocksDB or the in-mem Hashtable.
- An operator can acknowledge a checkpoint as soon as the changes in the log have reached the durable checkpoint storage.
- The state tables are persisted periodically, independent of the checkpoints. We call this the materialization of the state on the checkpoint storage.
- Once the state is materialized on checkpoint storage, the state changelog can be truncated to the corresponding point.
This approach mirrors what database systems do, adjusted to distributed checkpoints:
...