Page History

...

This KIP proposes to make new leader information available in the ProduceResponse and FetchResponse when the CurrentLeaderEpoch in the request is less than the leader epoch a newer leader is known by the receiving broker. This should help us eliminate the metadata refresh metadata refresh from the critical path of the Produce and Fetch requests, which tend to can be very latency-sensitive. This is an optimization mentioned in KIP-595, which uses a similar technique but only for the metadata log and the Fetch RPC.

...

The client will only accept the new leader information(LeaderId & LeaderEpoch) only if it advances its view of the new leader(i.e. new-leader’s Epoch should be greater than what client knows already) and use it in subsequent retries of the Produce & Fetch requests. On the client, it can happen that the subsequent metadata refreshes return stale leader information, if the latest metadata isn’t yet fully propagated to the entire cluster. The client will make sure that new leader information isn’t overridden by the stale leader’s information(again comparing LeaderEpochs). The , which is the existing behaviour of Kafka Java client does this already in Metadata::updateLatestMetadata.

Even though client will receive the new leader information in the ProduceResponse & FetchResponse when leader changes, but same as the existing behaviour of the Kafka Java client, it will request expedited metadata-refresh done asynchronously. Since leadership change will likely affect many partitions, so future requests to such partitions will benefit from this.

...