Page History

...

1. Each broker keeps in memory a timestamp index map - Map[TopicPartitionSegment, Map[TimestampByMinute, Offset]]
   1. The timestamp is on minute boundary
   2. The offset is the offset of the first message in the log segment that falls into a minute

2. Create a timestamp index file for each log segment. The entry in the file is as below:

   Code Block
   language java
   Time Index Entry => Timestamp Offset Timestamp => int64 Offset => int32

   So the timestamp index file will simply become a persistent copy of timestamp index map. Broker will load the timestamp map from the file on startup.

3. When a broker , (regardless leader or follower, ) receives a message, it does the following:
   1. Find which minute MIN the message with offset OFFSET falls in
   2. Check if MIN has already been in the in memory timestamp map for current log segment. If the timestamp does not exist, then the broker add [MIN->OFFSET] to both the in memory timestamp index map and the timestamp index file.
4. When a log segment is deleted, the broker:
   
   1. Remove the TopicPartitionSegment key from in memory map
   2. Remove the log segment timestamp index file

Comparison between Option 1 and Option 2

...