...

To address (2), the producer currently maintains an expiry threshold for every topic, which is used to remove a topic from the working set at a future time (currently hard-coded to 5 minutes, this should be modified to use metadata.max.age.ms). While this does work to reduce the size of the topic working set, the producer will continue fetching metadata for these topics in every metadata request for the full expiry duration. This logic can be made more intelligent by managing the expiry from when the topic was last used, enabling the expiry duration to be reduced to improve cases where a large number of topics are touched intermittently. To control behavior, a configuration variable topic.expiry.ms should be added to the producer configuration.Additionally, a second configuration variable topic.refresh.ms may be introduced to control the permitted staleness of a topic's metadata. Then, when topic metadata is fetched, only topics whose metadata is older than the refresh value need to be included in the request. Background, best-effort updates should be performed for topics with stale metadata, which would work towards reducing the number of topics included in a single topic metadata request.

Compatibility, Deprecation, and Migration Plan

...