THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
Change time based log rolling and retention to use timestamp
Currently time Time based log rolling and retention currently use the file creation time and last modified time. This does not work for newly created replicas because the time attributes of files is different from the true message append time. The following changes will address this issue.
- The time based log rolling will be based on the timestamp of the first message in the log segment file.
- The time based log retention will be based on the timestamp of the first last message in the log segment file.
...
Create another index file for each log segment with name SegmentBaseOffset.time.index to have index at minute level. The time index entry format is:
| Code Block | ||
|---|---|---|
| ||
Time Index Entry => Timestamp Offset Timestamp => int64 Offset => int32 |
The time index will be built based on the log index file. Every time when a new entry is inserted into log index file, we take a look at the timestamp of the message and if it falls into next minute, we insert an entry to the time index as well. The following table give the summary of memory consumption using different granularity. The number is calculated based on a broker with 3500 partitions.
...