...
Discussion thread: [DISCUSS] KIP-258: Allow to Store Record Timestamps in RocksDB
JIRA:
Jira | ||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|
|
Released: 1.2
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
...
In order to improve the provided stream processing semantics of – and to add new feature to – Kafka Streams, we want to be able to store record timestamps in KTables. This allows us to address multiple issues like
- store timestampe in KTables (KAFKA-6521)
- handling out-of-order data for source KTable (related to to KAFKA-5533)
- add TTL for KTables improve timestamp propagation for DSL operator (KAFKA-4212 and KAFKA-42736455)
- return the timestamp of the latest update in Interactive Queries (KAFKA-4304)
- improve timestamp propagation for DSL operator add TTL for KTables (KAFKA-64554212 and KAFKA-4273)
The challenging part of this KIP is to define a smooth upgrade path with the upgraded RocksDB format. There are some initial thoughts on KAFKA-3522 already. The rest of this doc will first focus on the public interface additions for the upgrade path. Storing timestamps in RocksDB is just a special case though, and we propose to allow for a generic upgrade path from any storage format A to any other storage format B. This include local storage in RocksDB as well as the underlying changelog topic.
...