...

As per the Tiered Storage feature introduced in KIP-405, the follower fetch protocol was modified for topics enabled with tiered storage. For such topics, the empty follower finds the offset and leader epoch, up to which the auxiliary state (i.e. leader epoch sequence, producer snapshot state) needs to be built from the leader. The follower then starts fetching the data from the leader starting from that offset. In the KIP, it was decided to use the local-log-start-offset as the offset for this purpose. Instead of the local-log-start-offset, we could also last-tiered-offset which would help the followers quickly catch up with the leader as it only needs to fetch the followers that are not moved to tiered storage. The disadvantage of using the last-tiered-offset is that this new follower would possess only a limited number of locally stored segments. Should it ascend to the role of leader, there is a risk of needing to fetch these segments from remote locations, potentially impacting broker performance.

...

The following diagram provides a visual representation of the leader topic partitonpartition's log offsets.

         Lx  = Local log start offset           L_z  = Local log end offset            L_y  = Last stable offset(LSO)

...

    Lz >= Ly >= Lx and Ly >= Ry >= Rx

Let us quicky quickly look at the current follower fetch protocol. The follower does the following:

• It requests data from the leader, beginning from its Fetch-Offset 
  • If the follower is empty, the Fetch-Offset is zero.
• The requested offset may or may not be a valid offset on the leader
  • Case 1: The requested offset is present on the leader
    • Leader The leader responds with the data to the follower
    • Follower receives the data, appends records to the local disk, updates its Log-End-Offset and its Fetch-Offset
    • Follower requests data beginning from the new fetch offset
  • Case 2: The requested offset is not present on the leader. There are two possible scenarios here:
    • Case 2.1: The requested offset is lower than the earliest offset the leader knows
      • Leader response with OFFSET_OUT_OF_RANGE error
      • Follower The follower fetches the earliest offset on the leader (Log-Start-Offset)
      • Follower updates its offsets
        • Log-Start-Offset → leader's Log-Start-Offset
        • Log-End-Offset → leader's Log-Start-Offset
        • Fetch-Offset → leader's Log-Start-Offset
      • Follower The follower requests data from the new fetch offset.
    • Case 2.2: The requested offset is greater than or equal to the earliest offset the leader knows, but the offset has moved to tiered storage.
      • Leader responds with OFFSET_MOVED_TO_TIERED_STORAGE error (also returns its Log-Start-Offset)
      • Follower fetches the leader's Local-Log-Start-Offset
      • Follower builds the remote log auxiliary states for the leader offsets in the range [Log-Start-Offset,  Local-Log-Start-Offset]
      • Follower updates its offsets
        • Log-Start-Offset → Leader's Log-Start-Offset
        • Log-End-Offset → Leader's Local-Log-Start-Offset
        • Fetch-Offset → Leader's Local-Log-Start-Offset
      • Follower The follower requests data from the new fetch offset.

...

• Leader responds with either OFFSET_OUT_OF_RANGE or OFFSET_MOVED_TO_TIERED_STORAGE error
• Follower fetches the earliest offset on the leader (Log-Start-Offset), if not available in the leader response
• Follower fetches the earliest offset on the leader that is waiting to be uploaded (Earliest-Pending-Upload-Offset). This is the next offset after the last-tiered-offset.
• Follwer Follower builds the remote log auxiliary states for the leader offsets in the range [Log-Start-Offset, Earliest-Pending-Upload-Offset)
• Follower updates its offsets
  • Log-Start-Offset → Leader's Log-Start-Offset
  • Log-End-Offset → Earliest-Pending-Upload-Offset
  • Fetch-Offset → Earliest-Pending-Upload-Offset
• Follower The follower requests data from the new fetch offset.

ReplicaFetcher Changes

On receiving an OFFSET_OUT_OF_RANGE or OFFSET_MOVED_TO_TIERED_STORAGE error, ReplicaFetcher does the following:

• Check if the follower replica is empty and if the feature to use last-tiered-offset is enabled.

Follower Fetch Scenarios

Scenario 1: New Empty Follower

...

Continue fetching from the leader

Scenario 2: Out-of-sync follower catching up

An out-of-sync follower (Broker B) has local data up to offset 3

Step 1:

Step 2:

We shall also add a new API for the follower to be able to fetch the pending-upload-offset from the leader. The leader already keeps track of the highest offset that has been uploaded to the remote storage.

...

• What impact (if any) will there be on existing users?
• If we are changing behavior how will we phase out the older behavior?
• If we need special migration tools, describe them here.
• When will we remove the existing behavior?

Test Plan

Describe in few sentences how the KIP will be tested. We are mostly interested in system tests (since unit-tests are specific to implementation details). How will we know that the implementation works as expected? How will we know nothing broke?Unit Tests and Integration Tests for the various follower fetch scenarios.

Rejected Alternatives

If there are alternative ways of accomplishing the same thing, what were they? The purpose of this section is to motivate why the design is the way it is and not some other way.