...

Currently, when a new consumer or a fallen-off consumer requires fetching messages from a historical periodwhile ago, and those messages are no longer present in the Kafka broker's local storage, the broker must download the message from the remote tiered storage and subsequently transfer the data back to the consumer. This process involves two data transfers, resulting in increased network latency and transfer costs. Critically, when a broker downloads the data segment from remote storage and stores it locally, it imposes additional IOPS on the network and local disk capacity. Furthermore, this action negatively impacts the page cache established for messages closer to the end of the message queue.

With the data segment file already present on remote tiered storage, there is no reason why the consumer cannot directly fetch them from remote storage without affecting the broker performance.  To address this gap, we propose here to allow the consumers to fetch from the remote tiered storage.

This feature will also be highly beneficial for serving large backfill requests from consumer clients from data analytics and streaming applications such as Pinot or Flink.

Proposed Changes

Know the location of remote log segment file

...