Page History

...

With KIP-33, the brokers can now search messages by timestamp accurately. To maintain the backward backwards compatibility, we did not change the behavior of ListOffsetRequest v0, which still return a list of base offsets of the segments whose last modification time is smaller than or equals to the target time. In this KIP, we will introduce ListOffsetRequest v1 to provide more accurate search based on timestamp.

Part 2: Add a search message by timestamp method to o.a.k.c.consumer.Consumer

...

1. From time to time, applications may need to reconsume the messages for a cerntain period, so they will need to rewind the offsets back to, say 6 hours ago, and reconsume all the messages after that.
2. In a multi-datacenter enviroment, users may have different Kafka clusters in each datacenter for disater recovery. If one of the datacenter failed, the applications may need to switch the consumption from one data center to another datacenter. Because the offset between two different Kafka clusters are independent, users cannot use the offsets from the failed datacenter to consume from the DR datacenter. In this case, searching by timestamp will help because the messages should have the same timestamp if users are using CreateTime. Even if users are using LogAppendTime, a more granular search based on timestamp can still help reduce the amount of messages to be reconsumed.

Another related feature missing in KafkaConsumer is the access of partitions' high watermark. Typically, users only need the high watermark in order to get the per partition lag. This seems more suitable to be exposed through the metrics.

...