Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Currently, followers build the auxiliary state (i.e. leader epoch sequence, producer snapshot state) when they fetch the messages from the leader by reading the message batches. Incase of tiered storage, follower finds the offset and leader epoch up to which the auxiliary state needs to be built from the leader. After which, followers start fetching the data from the leader starting from that offset. That offset can be local-log-start-offset or nextlast-localtiered-offset. Local-log-start-offset is the log start offset of the local storage. Next local offset Last-tiered-offset  is the offset up to which the segments are copied to remote storage. We will describe pros and cons of choosing these segments.

nextlast-localtiered-offset

  • The advantage with this option is that followers can catch up quickly with the leader as the segments that are required to be fetched by followers are the segments that are not yet moved to remote storage.  
  • One disadvantage with this approach is that followers may have a few local segments than the leader. When that follower becomes a leader then the existing followers will truncate their logs to the leader's local log-start-offset. 

...

So, we need to add a respective ReplicaState for building auxiliary state which can be called `BuildingRemoteLogAuxBuildingRemoteLogAuxState`. Fetcher thread processes this state also in every run like as it does for Fetching and Truncating states.

...