Versions Compared

Old Version 10

changes.mady.by.user Abhijeet Kumar

Saved on

compared with

New Version 11

changes.mady.by.user Abhijeet Kumar

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

  • Type: Boolean
  • Mode: Dynamically configurable as cluster-default for all brokers in the cluster.
  • Description: Whether the last tiered offset should be used as the start offset for bootstrapping an empty follower
  • Default value: False

Proposed Changes

High-Level Design

The following diagram provides a visual representation of the leader topic partiton's log offsets.

...

  • Leader responds with either OFFSET_OUT_OF_RANGE or OFFSET_MOVED_TO_TIERED_STORAGE error
  • Follower fetches the earliest offset on the leader (Log-Start-Offset), if not available in the leader response
  • Follower fetches the earliest offset on the leader that is waiting to be uploaded (Earliest-Pending-Upload-Offset). This is the next offset after the last-tiered-offset.
  • Follwer builds the remote log auxiliary states for the leader offsets in the range [Log-Start-Offset, Earliest-Pending-Upload-Offset)
  • Follower updates its offsets
    • Log-Start-Offset → Leader's Log-Start-Offset
    • Log-End-Offset → Earliest-Pending-Upload-Offset
    • Fetch-Offset → Earliest-Pending-Upload-Offset
  • Follower requests data from the new fetch offset.

Follower Fetch Scenarios

Scenario 1: New Empty Follower

Step 1:

...

Broker A (Leader)Broker B (Follower)Remote StorageRL Metadata Storage

3: msg 3 LE-1

4: msg 4 LE-1

5: msg 5 LE-2

6: msg 6 LE-2

7: msg 7 LE-3 (HW)


leader_epochs:

LE-0, 0

LE-1, 3

LE-2, 5

LE-3, 7

  1. Fetch LE-1, 0
  2. Receives OMTS
  3. Receives ELO 3, LE-1
  4. Fetches PLOEPUO
  5. Receives PLO EPUO 6, LE-2
  6. Fetch remote segment info and build local leader epoch sequence until ELO


leader_epochs:

LE-0,0

LE-1,3

LE-2,5


seg-0-2, uuid-1

  log:

  0: msg 0 LE-0

  1: msg 1 LE-0

  2: msg 2 LE-0

  epochs:

  LE-0, 0


seg 3-5, uuid-2

  log:

  3: msg 3 LE-1

  4: msg 4 LE-1

  5: msg 5 LE-2

  epochs:

  LE-0, 0

  LE-1, 3

  LE-2, 5  

seg-0-2, uuid-1

segment epochs:

LE-0, 0


seg-3-5, uuid-2

segment epochs:

LE-1, 3

LE-2, 5

...

Continue fetching from the leader

Broker A (Leader)Broker B (Follower)Remote StorageRL Metadata Storage

3: msg 3 LE-1

4: msg 4 LE-1

5: msg 5 LE-2

6: msg 6 LE-2

7: msg 7 LE-3 (HW)


leader_epochs:

LE-0, 0

LE-1, 3

LE-2, 5

LE-3, 7

Fetch from PLO to HW

6: msg 6 LE-2

7: msg 7 LE-3 (HW)


leader_epochs:

LE-0,0

LE-1,3

LE-2,5

LE-3,7


seg-0-2, uuid-1

  log:

  0: msg 0 LE-0

  1: msg 1 LE-0

  2: msg 2 LE-0

  epochs:

  LE-0, 0


seg 3-5, uuid-2

  log:

  3: msg 3 LE-1

  4: msg 4 LE-1

  5: msg 5 LE-2

  epochs:

  LE-0, 0

  LE-1, 3

  LE-2, 5  

seg-0-2, uuid-1

segment epochs:

LE-0, 0


seg-3-5, uuid-2

segment epochs:

LE-1, 3

LE-2, 5

Scenario 2: Out-of-sync follower catching up



We shall also add a new API for the follower to be able to fetch the pending-upload-offset from the leader. The leader already keeps track of the highest offset that has been uploaded to the remote storage.

...