Page History

...

1. When a new log segment is created, the broker will create a time index file for the log segment. 
2. The default initial / max size of the time index files is the same as the offset index files. (time index entry is 1.5x of the size of offset index entry, user should set the configuration accordingly).
3. Each log segment maintains the largest timestamp so far in that segment. The initial value of the largest timestamp is -1 for a newly created segment.
4. When broker receives a message, if the message is not rejected due to timestamp exceeds threshold, the message will be appended to the log. (The timestamp will either be LogAppendTime or CreateTime depending on the configuration)
5. When broker appends the message to the log segment, if an offset index entry is inserted, it will also insert a time index entry if the max timestamp so far is greater than the timestamp in the last time index entry.
   • For message format v0, the timestamp is always -1, so no time index entry will be inserted when message is appended.
6. When a new log segment is rolled out. A time index entry will be inserted into the time index to ensure the last time index entry. 
   1. If largest timestamp in the segment is non-negative (at least one message has a timestamp), the entry will be (largest_timestamp_in_the_segment -> base_offset_of_the_next_segment)
   2. If largest timestamp in the segment is -1 (No message in the segment has a timestamp),  the entry will be (last_modification_time_of_the_segment -> base_offset_of_next_segment) 

...

Broker startup

On broker startup, The the latest timestamp is needed for the next log index append. The broker will find the largest timestamp by looking at the last inserted time index entry and scan from there till the log end. Broker will only do this if message.format.version is greater than on or after 0.10.0. Otherwise the broker will skip reloading loading the largest timestamp.

Log Truncation

...