Versions Compared

Old Version 13

changes.mady.by.user Ying Zheng

Saved on

compared with

New Version 14

changes.mady.by.user Ying Zheng

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

After successfully copied a segment file to remote storage, RLM will append a set of index entries to 3 local index files: remotelogindex, remoteoffsetindex, remotetimeindex. These index files are rotated by RLM at a configurable time interval (or a configurable size).

(active segment)

{log.dirs}/{topic-partition}/0000002400013.index

{log.dirs}/{topic-partition}/0000002400013.timeindex

{log.dirs}/{topic-partition}/0000002400013.log


(inactive segments)

{log.dirs}/{topic-partition}/0000002000238.index

{log.dirs}/{topic-partition}/0000002000238.timeindex

{log.dirs}/{topic-partition}/0000002000238.log

{log.dirs}/{topic-partition}/0000001600100.index

{log.dirs}/{topic-partition}/0000001600100.timeindex

{log.dirs}/{topic-partition}/0000001600100.log


(active remote segment)

{log.dirs}/{topic-partition}/0000001000121.remoteoffsetindex

{log.dirs}/{topic-partition}/0000001000121.remotetimeindex

{log.dirs}/{topic-partition}/0000001000121.remotelogindex


(inactive remote segments)

{log.dirs}/{topic-partition}/0000000512002.remoteoffsetindex

{log.dirs}/{topic-partition}/0000000512002.remotetimeindex

{log.dirs}/{topic-partition}/0000000512002.remotelogindex



Each index entry of the remotelogindex file contains the information of a sequence of records in the remote log segment file. The format of a remotelogindex entry:

magic: int16 (current magic value is 0)

length: int16 (length of this entry)

crc: int32 (checksum of this entry)

startOffset

firstOffset: int64 (the Kafka offset of the 1st record)

lastOffset: int64 (the Kafka offset of the last record)

firstTimestamp: int64

maxTimestamp

lastTimestamp: int64

dataLength: int32 (length of the data)

rdiLength: int16

rdi: byte[] (Remote data identifier)


RDI (Remote data identifier) is the "pointer" or "URI" of the remote data. The format of RDI depends on the implementation. For example, RDI can be HDFS file path and offset, or S3 key and offset. When reading the remote records, RLM will use RDI to retrieve the remote data.

...