Page History

...

RLM creates tasks for each leader or follower topic partition:

• RLM Leader Task -
  • It checks for rolled over LogSegments (which have the last message offset less than last stable offset of that topic partition) and copies them along with their remote offset/time indexes to the remote tier.
  RLM creates an index file, called RemoteLogSegmentIndex, per topic-partition to track remote LogSegments. These indexes are described in detail here.
  • It also serves the fetch requests for older data from the remote tier. Local logs are not
  cleanedup
  • cleaned up till those segments are copied successfully to remote even though their retention time/size is reached.
  • We proposed an approach to creating a RemoteLogSegmentIndex, per topic-partition to track remote LogSegments. These indexes are described in more detail here. This allows having a larger index interval of remote log segments instead of a large number of short index files. It also supports encrypted segments by encrypting individual record batch and build the respective indexes as the local segment will not be useful. The initial version of this approach is implemented in PR. We want to proceed with local log indexes in the initial version and we may want to go with remote log indexes in later versions.  
• RLM Follower Task RLM Follower Task -
  • It keeps track of the segments and index files on remote tier by looking into RemoteLogMetdataManager. RLM follower can also serve reading old data from the remote tier.

  Core Kafka changes

  To satisfy the goal of keeping Kafka changes minimal when RLM is not configured, Kafka behavior remains unchanged for existing users.

  ...