...
After successfully copied a segment file to remote storage, RLM will append a set of index entries to 3 local index files: remoteLogIndex, remoteOffsetIndex, remoteTimeIndex. These index files are rotated by RLM at a configurable time interval (or a configurable size).
(active segment) {log.dirs}/{topic-partition}/0000002400013.index {log.dirs}/{topic-partition}/0000002400013.timeindex {log.dirs}/{topic-partition}/0000002400013.log (inactive segments) {log.dirs}/{topic-partition}/0000002000238.index {log.dirs}/{topic-partition}/0000002000238.timeindex {log.dirs}/{topic-partition}/0000002000238.log {log.dirs}/{topic-partition}/0000001600100.index {log.dirs}/{topic-partition}/0000001600100.timeindex {log.dirs}/{topic-partition}/0000001600100.log (active remote segment) {log.dirs}/{topic-partition}/0000001000121.remoteOffsetIndex {log.dirs}/{topic-partition}/0000001000121.remoteTimeIndex {log.dirs}/{topic-partition}/0000001000121.remoteLogIndex (inactive remote segments) {log.dirs}/{topic-partition}/0000000512002.remoteOffsetIndex {log.dirs}/{topic-partition}/0000000512002.remoteTimeIndex {log.dirs}/{topic-partition}/0000000512002.remoteLogIndex |
Each index entry of the remoteLogIndex file contains the information of a sequence of records in the remote log segment file. The format of a remoteLogIndex entry:
magic: int16 (current magic value is 0) length: int16 (length of this entry) crc: int32 (checksum from firstOffset to the end of this entry) firstOffset: int64 (the Kafka offset of the 1st record) lastOffset: int64 (the Kafka offset of the last record) firstTimestamp: int64 lastTimestamp: int64 dataLength: int32 (length of the remote data) rdiLength: int16 rdi: byte[] (Remote data identifier) |
todo: We may change this format to have magic and crc for a batch of entries instead of having them for each entry.
...
Copy the log segment files that are 1)
inactive and
2) the offset range is not covered by the segments on the remote storage.
If multiple log segment files are ready, they are copied to remote storage one by one, from the earliest to the latest.
...