...

The change is backwards compatible with previous versions.  The new parameter RemoteLogSegmentLocationRequested in consumer FetchRequest is false by default to not to change the FetchResponse behavior. And If the receiving broker does not support this feature, it will resort to the old behavior of fetching old messages from remote tiered storage and sending them back to the consumer client.

Rejected Alternatives

1. Having the consumer subscribes the tiered storage metadata topic: __remote_log_metadata, filters to the partition it interests in, parses each metadata message and tracks the state change;  We don’t recommend this approach because:
   1. __remote_log_metaddata topic is a congested topic in tiered storage which contains metadata for all the tiered storage enabled topic partitions and each metadata needs to go through several state changes.  The topic includes the metadata history of all the remote log segment from day 1 even if the segment file itself is already removed from tiered storage; There are lot of reading/filtering and caching needs to happen on the consumer side;
   2. With many programming languages to support on the consumer side and given the trend of moving complex consumer logic to the broker side, we feel it is more natural to have the broker to keep track of the remote tiered storage segment file and exchange the info with the consumer.  This is similar to the design choice (whether consumer or the broker keep tracks of preferred read replica) during KIP-392: Allow consumers to fetch from closest replica design where it also favors to have the broker exchange the preferred read replica information with the consumer.
2. Launch new brokers to serve the large backfill read use cases.  When the consumer asks the leader broker to serve FetchRequest, the leader broker will respond with those new brokers' ids in FetchResponse so the consumer will subsequently asks those new brokers to serve FetchRequest.  This is very similar to how leader broker responded with preferredReadReplica in KIP-392.  Those new brokers can still download log segments from remote storage before they can serve FetchRequest.  In this approach, the network latency/cost is not avoided, but we can keep the current running brokers not being affected by reads from large backfill consumers.  This approach has minimal code changes needed on the consumer side, but it has the following drawbacks:
   1.  In order not to affect the performance of the current running brokers, we need to group the brokers into different pools (a pool to serve the normal producing/consumer use cases, a pool to serve the backfill or other maintenance tasks), this is a new concept for Kafka community;
   2. The user needs to manually start/stop those new backfill-dedicated brokers before/after backfill task;
   3. In order to quickly bootstrap those new brokers to ready-to-serve state, we need KIP-1023: Follower Fetch from tiered offset to be implemented first.  With KIP-1023, the new brokers only needs to replicate a few minutes of data from leader broker to be considered in-sync (the rest of the data can be downloaded later lazily from tiered storage);
   4. Network cost/latency related to 2-hop data transfer (from remote storage to broker, from broker to consumer) is still there.