...

A new dynamic broker configuration: fetch.remote.max.wait.ms will be added and the delayed remote fetch purgatory will wait up to this timeout to fetch the data from the remote storage.

The consumer config fetch.max.wait.ms document will be updated to denote that it applies only to the local storage:

The maximum amount of time the server will block before answering the fetch request when it is reading near to the tail of the partition (high-watermark) if there isn't sufficient data to immediately satisfy the requirement given by fetch.min.bytes

Proposed Changes

Remote storage read latencies are non-deterministic. Suppose the user configures 20 remote-log-reader threads, and it takes 100 ms to serve one request. When a backfill job runs and reads data from the head of the log for multiple partitions (let's say 1000), the remote-fetch requests get queued, potentially exceeding the default timeout of 500 ms. Additionally, the time taken to serve the P99 remote storage fetch requests can extend into seconds. We propose introducing a new timeout parameter, fetch.remote.max.wait.ms, to offer users the option to configure the timeout based on their workload.

...