Page History

...

• It requests data from the leader, beginning from its Fetch-Offset 
  • If the follower is empty, the Fetch-Offset is zero.
• The requested offset may or may not be a valid offset on the leader
  • Case 1: The requested offset is present on the leader
    • The leader responds with the data to the follower
    • Follower receives the data, appends records to the local disk, updates its Log-End-Offset and its Fetch-Offset
    • Follower requests data beginning from the new fetch offset

                     This case is also true when the leader has uploaded the segments, but hasn't cleared the segments from the local disk yet. In such a case, the requested offset (offset 0) may still be available on the leader and the leader will respond with the data to the follower.

• • Case 2: The requested offset is not present on the leader. There are two possible scenarios here:
    • Case 2.1: The requested offset is lower than the earliest offset the leader knows
      • Leader response with OFFSET_OUT_OF_RANGE error
      • The follower fetches the earliest offset on the leader (Log-Start-Offset)
      • Follower updates its offsets
        • Log-Start-Offset → leader's Log-Start-Offset
        • Log-End-Offset → leader's Log-Start-Offset
        • Fetch-Offset → leader's Log-Start-Offset
      • The follower requests data from the new fetch offset.
    • Case 2.2: The requested offset is greater than or equal to the earliest offset the leader knows, but the offset has moved to tiered storage.
      • Leader responds with OFFSET_MOVED_TO_TIERED_STORAGE error (also returns its Log-Start-Offset)
      • Follower fetches the leader's Local-Log-Start-Offset
      • Follower builds the remote log auxiliary states for the leader offsets in the range [Log-Start-Offset,  Local-Log-Start-Offset]
      • Follower updates its offsets
        • Log-Start-Offset → Leader's Log-Start-Offset
        • Log-End-Offset → Leader's Local-Log-Start-Offset
        • Fetch-Offset → Leader's Local-Log-Start-Offset
      • The follower requests data from the new fetch offset.

We make the following changes to the follower-fetch protocol when the requested offset is not present on the leader (Case 2). These changes apply only when the follower is empty.

...

• The RLM task on the leader runs at frequent intervals to find new log segments that have become eligible for upload.
• When the task uploads a segment, it also updates the highest offset which has been uploaded to the remote storage.
• If the leader is newly elected, it may not have this information yet and , but will learn about the highest offset on remote only when a new segment is uploaded.We will modify the RLM task behavior to update the highest offset information even if there are no new segments to upload. This ensures that the RLM task has this information ready even it is not able to find new segments eligible for upload.the corresponding RLM Task runs for the first time.
• The new API returns the offset following the highest offset on the remote as the response to fetch the Earliest-Pending-Upload-Offset

...

• Check if the follower replica is empty and if the feature to use last-tiered-offset is enabled. If the check fails, continue with the old behaviour.
• Otherwise, do the following:
  • Fetch the leader's log-start-offset
  • Fetch the leader's Earliest-Pending-Upload-Offset using the API discussed in the previous section
  • If the Earliest-Pending-Upload-Offset is unknown (API returned -1), there are two possible cases:
    • No segments for the partition have been uploaded to the remote storage yet
      • It can be confirmed by checking if the leader's Log-Start-Offset is the same as the Leader's Local-Log-Start-Offset.
      • If confirmed, we use the Local-Log-Start-Offset as the Earliest-Pending-Upload-Offset
    • Segments are uploaded to the remote storage but the leader does not know yet.
      • In this case, the leader will eventually learn about the Earliest-Pending-Upload-Offset.
      • The ReplicaFetcher should throw an exception so that the partition can be retried after some time (while the leader learns about the offset)
  • The highest offset on the remote storage may be much lower than the leader's Log-Start-Offset. Hence, the Earliest-Pending-Upload-Offset returned by the leader may also be less than the leader's Log-Start-Offset. This may happen because of the following situation:
    • Tiered Storage was enabled for the topics and log segments were uploaded to the remote storage
    • Later, tiered storage was disabled for the topic and more messages were produced on the topic
    • Some time has elapsed and now the leader's Log-Start-Offset is much ahead of the highest offset on the remote storage.
    • Now tiered storage is enabled for the topic, but no segments have been uploaded yet because no new segments are eligible to be uploaded yet 

                      Whenever this happens (Earliest-Pending-Upload-Offset < Log-Start-Offset), it simply means that they there are no valid log segments on the remote yet (otherwise, the leader would return an offset >= log-start-offset). Hence the ReplicaFetcher simply truncates its log and starts replicating all data the leader has locally.

...