Page History

...

For any fetch requests, ReplicaManager will proceed with making a call to readFromLocalLog, if this method returns OffsetOutOfRange exception it will delegate the read call to RemoteLogManager.readFromRemoteLog and returns the LogReadResult. If the fetch request is from a consumer, RLM will read the data from remote storage, and return the result to the consumerMore details are explained in the ThreadPools section.

Follower Requests/Replication

For follower fetch, the leader only returns the data that is still in the leader's local storage. If a LogSegment copied into Remote Storage remote storage by a Leader Brokerleader broker, Follower follower doesn't need to copy this segment which is already present in Remote Storageremote storage. Instead, a follower will retrieve the information of the segment from remote storage. If a Replica becomes a Leaderleader, It can still locate and serve data from Remote remote storage.

Thread Pools

Remote storage (e.g. S3 / HDFS) is likely to have higher I/O latency and lower availability than local storage.

...

When handling consumer fetch request, if the required offset is in remote storage, the request is added into "RemoteFetchPurgatory", to handle timeout. RemoteFetchPurgatory is an instance of kafka.server.DelayedOperationPurgatory, and is similar to the existing produce/fetch purgatories. At the same time, the request is put into the task queue of "remote storage fetcher thread pool".

...