DUE TO SPAM, SIGN-UP IS DISABLED. Goto Selfserve wiki signup and request an account.
...
- Leader responds with either OFFSET_OUT_OF_RANGE or OFFSET_MOVED_TO_TIERED_STORAGE error
- Follower fetches the earliest offset on the leader (Log-Start-Offset), if not available in the leader response
- Follower fetches the last tierd earliest offset on the leader (Last-Tieredthat is waiting to be uploaded (Pending-Upload-Offset). This is the next offset after the last-tiered-offset.
- Follwer builds the remote log auxiliary states for the leader offsets in the range [Log-Start-Offset, LastPending-TieredUpload-Offset])
- Follower updates its offsets
- Log-Start-Offset → Leader's Log-Start-Offset
- Log-End-Offset → LastPending-TieredUpload-Offset + 1
- Fetch-Offset → LastPending-TieredUpload-Offset + 1
- Follower requests data from the new fetch offset.
| Broker A (Leader) | Broker B (Follower) | Remote Storage | RL Metadata Storage |
|---|---|---|---|
3: msg 3 LE-1 4: msg 4 LE-1 5: msg 5 LE-2 6: msg 6 LE-2 7: msg 7 LE-3 (HW) leader_epochs LE-0, 0 LE-1, 3 LE-2, 5 LE-3, 7 |
| seg-0-2, uuid-1 log: 0: msg 0 LE-0 1: msg 1 LE-0 2: msg 2 LE-0 epochs: LE-0, 0 seg 3-5, uuid-2 log: 3: msg 3 LE-1 4: msg 4 LE-1 5: msg 5 LE-2 epochs: LE-0, 0 LE-1, 3 LE-2, 5 | seg-0-2, uuid-1 segment epochs LE-0, 0 seg-3-5, uuid-2 segment epochs LE-1, 3 LE-2, 5 |
We shall also add a new API for the follower to be able to fetch the lastpending-tieredupload-offset from the leader. The leader already keeps track of the highest offset that has been uploaded to the remote storage.
- The RLM task on the leader runs at frequent intervals to find new log segments that have become eligible for upload.
- When the task uploads a segment, it also updates the highest offset which has been uploaded to the remote storage.
- If the leader is newly elected, it may not have this information yet and will learn about the highest offset on remote only when a new segment is uploaded.
- We will modify the RLM task behaviour to update the highest offset information even if there are no new segments to upload. This ensures that the RLM task has this information ready even it is not able to find new segments eligible for upload.
- The new API returns the offset following the highest offset on the remote as the response to fetch lastpending-tieredupload-offset
Compatibility, Deprecation, and Migration Plan
...