Page History

...

...

...

JIRA: 

...

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

...

• Stateful operators write the state changes to that log (logging the state), in addition to applying them to the state tables in RocksDB or the in-mem Hashtable.
• An operator can acknowledge a checkpoint as soon as the changes in the log have reached the durable checkpoint storage.
• The state tables are persisted periodically, independent of the checkpoints. We call this the materialization of the state on the checkpoint storage.
• Once the state is materialized on checkpoint storage, the state changelog can be truncated to the corresponding point.

This approach mirrors what database systems do, adjusted to distributed checkpoints:

...