...

1. The log retention will be changed to base on the time index of a log segment instead of basing on the last modification time of the log segment file.
2. The time based log rolling will have the following change: The log segment will roll out when log.roll.ms has elapsed since the largest timestamp of the messages in the log segment.

This KIP will indroduce introduce another index file for each log segment. So the number of open file handlers will increased increase by about 1/3.

Proposed Changes

...

1.  The broker will enforce log retention using time index. Given what we do in step 1, the behavior is
   1. For segments only having messages whose versions are before 0.10.0, the entry with last modification time in the time index will be used for retention.
   2. For segments having at least one message with version 0.10.0, the max timestamp of the messages will be used for log retention.
2. The broker will enforce the log rolling time when there is at least one message with format version 0.10.0 inserted, otherwise the log rolling is still based on segment create time.
3. Searching by timestamp will behave the same as before because if there is no message with timestamp, the time index will only contain one entry with last_modification_time. If there are messages with timestamp inserted, those messages will be indexed for search.

...