THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
Create another index file for each log segment with name SegmentBaseOffset.time.index. The granularity of the index is defined by time.index.interval.ms configuration.
The time index entry format is:
| Code Block | ||
|---|---|---|
| ||
Time Index Entry => Timestamp Offset Timestamp => int64 Offset => int32 |
The time index granularity does not change the actual timestamp searching granularity. It only affects the time needed for searching. The way it works will be the same as offset search - find the closet timestamp and corresponding offset, then start the linear scan over the log until find the target message. Although the granularity is configurable, it is recommended to have a minute level granularity because timestamp based search is usually rare so it probably does not worth investing significant amount of memory in it.
The following table give the summary of memory consumption of one day using different granularity. The number is calculated based on a broker with 3500 partitions.
...
- When broker receives a message, if the message is not rejected due to timestamp exceeds threshold, the message will be appended to the log.
- The timestamp will either be LogAppendTime or CreateTime depending on the configuration.
- The broker checks If the message timestamp . If it falls into a new time.index.interval bucket which is greater than the previous appended index entry, the broker appends a new time index entry to the time index with the new time.index.interval bucket timestamp.
- When a log segment is closed, if the message with largest timestamp is in this closed segment, the broker will insert a time index entry to the time index of the closed log segment. The time index entry points to the that message with largest timestamp in the closed log segment if the largest timestamp ever seen is in the closed segment.
- When a new log segment is created, the broker will insert a time index entry to the time index of the new log segment when the first message whose timestamp falls into a new time.index.interval bucket is appended to the log segment.
- It is possible that a log segment does not have any time index entry if all the messages has smaller timestamp than the previous log segments. In that case the time based index would be empty.
...