...

This KIP proposes making writes to the state stores transactional, so that they atomically commit only after the corresponding changes are committed to the changelog topic. As a result, Streams applications configured with EOS will no longer need to wipe the state stores on crash failure. Transactional writes can also be used in combination with ALOS to reduce duplication.

Public Interfaces

Changed:

...

This KIP proposes introducing an enabled by default feature flag (statestore.transactional.enabled)  that makes writes to state stores transactional, meaning they apply atomically after the applications commits to the changelog topic. The transactional guarantees come with non-zero overhead, so the feature flag acts as a safety switch if Kafka Streams users decide to opt-out. This feature presents a trade-off between increased read, write, and memory amplifications and time to recover from crash failure in the case of EOS. As for ALOS, the system benefits from fewer duplicated records.

When the feature flag is on, and the state store supports transactions (checked via new boolean method StateStore#transactional), it applies changes atomically only after they were committed to the changelog topic. Transactional state stores do not delete the checkpoint file and the underlying data in the case of EOS. Instead, they roll forward from the changelog topic on recovery. When the feature flag is off or the store does not support transactions, EOS recovery behaves as it does now - the task will delete the checkpoint file and recover from the changelog.

...