...
For upgraded consumers, they can handle both V0 and V1.
Rejected Alternatives
Add a timestamp field to log index entry
The most straight forward approach to have a time index is to let the log index files have a timestamp associate with each entry.
Code Block | ||
---|---|---|
| ||
Log Index Entry => Offset Position Timestamp
Offset => int32
Position => int32
Timestamp => int64 |
Having a LogAppendTime only.
Because the index entry size become 16 bytes instead of 8 bytes. The index file size also needs to be doubled. As an example, one of the broker we have has ~3500 partitions. The index file took about 16GB memory. With this new format, the memory consumption would be 32GB.