Page History

...

Discussion thread: [DISCUSS] KIP-258: Allow to Store Record Timestamps in RocksDB

JIRA: 

Released: 1.2

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

...

In order to improve the provided stream processing semantics of – and to add new feature to – Kafka Streams, we want to be able to store record timestamps in KTables. This allows us to address multiple issues like

• store timestampe in KTables (KAFKA-6521)
• handling out-of-order data for source KTable (related to to KAFKA-5533)
• add TTL for KTables improve timestamp propagation for DSL operator (KAFKA-4212 and KAFKA-42736455)
• return the timestamp of the latest update in Interactive Queries (KAFKA-4304)
• improve timestamp propagation for DSL operator add TTL for KTables (KAFKA-64554212 and KAFKA-4273)

The challenging part of this KIP is to define a smooth upgrade path with the upgraded RocksDB format. There are some initial thoughts on KAFKA-3522 already. The rest of this doc will first focus on the public interface additions for the upgrade path. Storing timestamps in RocksDB is just a special case though, and we propose to allow for a generic upgrade path from any storage format A to any other storage format B. This include local storage in RocksDB as well as the underlying changelog topic.

...