Page History

...

• It will delegate copy and read of these segments to a pluggable storage manager(viz RemoteStorageManager) implementation.

RLM has two modescreates tasks for each leader or follower topic partition:

• RLM Leader - In this mode, RLM is the leader for a topic-partition, Task- It checks for rolled over LogSegments (which have last message offset less than last stable offset of that topic partition) and copies them along with their remote offset/time indexes to the remote tier. RLM creates an index file, called RemoteLogSegmentIndex, per topic-partition to track remote LogSegments. These indexes are described in detail later here. It also serves the fetch requests for older data from the remote tier. Local logs are not cleanedup till those segments are copied successfully to remtoe remote even though their retention time/size is reached.
• RLM Follower Task - In this mode, RLM keeps track of the segments and index files on remote tier and updates its RemoteLogSegmentIndex file per topic-partition. RLM follower does not serve reading old data from the remote tier. Local logs are not cleanedup till their remote log indexes are copied locally from remote storage even though their retention time/size is reached.

...

• Core Kafka starts RLM service if tiered storage is configured
• When an offset index is not found, if RLM is configured, the read request is delegated to RLM to serve the data from the remote tier.

Serving Data from Remote Storage

Anchor
rdi-format rdi-format

For each topic partition that has RLM configured, RLM leader for a topic partition copies log segments which have last message offset less than last stable offset of that topic partition to remote storage. The active segment file (the last segment file of each partition, to which the new records are appending) is never shipped to remote storage.

After successfully copied a segment file is copied to remote storage, RLM will append a set of index entries to 3 local index files: remoteLogIndex, remoteOffsetIndex, remoteTimeIndex. These index files are rotated by RLM at a configurable time interval (or a configurable size).

...

When the follower discovers a new segment in remote storage, it will retrieve the index entries from remote storage and creates local remote log index file locally. The RemoteOffsetIndex file and RemoteTimestampIndex file are also created accordingly.

...