Page History

...

Followers fetch offsets and truncate their local logs if needed with the current mechanism based on leader's local-logStartOffset.

Leaders will send a new error code `OFFSET_MOVED_TO_TIERED_STORAGE` to followers if the fetch offset is not available in the local log. In this case, the follower needs to get auxiliary state of the remote og segments, which is leader epochs (possibly producer-snapshot-ids too). This can be done in two ways.

• introduce a new protocol (or API) to fetch these this state from the leader partition.
• fetch these this state from the remote storage. 

Latter is preferred here as remote storage can have this data state and it is simpler without introducing a new protocol with the leader.  

...

So, we need to add respective ReplicaStates for both which can be called `FetchingRemoteLogMetadata` and `FetchingRemoteLogAuxiliaryState`. Fetcher thread also processes both of these states in every run.  FetchingRemoteLogMetadata state checks whether RLMM on the follower was able to cacthup to receive the remote log segment metadata of the desired fetch offset. If it has a value then the state is moved to FetchingRemoteLogAuxiliaryState by submitting a task to get the state from RemoteStorageManager using RLM thread pool. If the FetchingRemoteLogAuxiliaryState already receives the state then the state is moved to Fetching with the fetch offset as the remote log segment's endOffset+1.

Transactional support

RemoteLogManager COPIES copies transaction index along with the respective log segment. This is used by the followers to return aborted transactions in fetch requests with isolation level as READ_COMMITTED. We may not need to copy producer-id-snapshot as we are copying only segments earlier to log-stable-offset. 

Consumer Fetch Requests

...