...

In concert, these factors amplify the negative effects of each other, and improvements should be made in order to alleviate topic scalability issues.

Public Interfaces

...

Adds producer configuration flag metadata.expiry.ms (default: 5 minutes) to control topic expiry duration.

/** <code>metadata.expiry.ms</code> */
public static final String METADATA_EXPIRY_MS_CONFIG = "metadata.expiry.ms";
private static final String METADATA_EXPIRY_MS_DOC =
        "Controls how long the producer will cache metadata for a topic that's not being accessed. " +
        "If the elapsed time since a topic was last produce to exceeds the metadata expiry duration, " +
        "then the topic's metadata is forgotten and the next access to it will force a metadata " +
        "fetch request.";

...
    .define(METADATA_EXPIRY_MS_CONFIG,
            Type.LONG,
            5 * 60 * 1000,
            atLeast(1000),
            Importance.MEDIUM,
            METADATA_EXPIRY_MS_DOC)

Proposed Changes

The proposal is to resolve (2), which should reduce the cost of (1) considerably.

The producer has two values of interest: an expiry threshold for topic metadata, which is used to remove an unused topic from the working set at a future time (currently hard-coded to 5 minutes), and a metadata refresh threshold, which is used to periodically refresh topic metadata (defined by metadata.max.age.ms). While seemingly similar, these two thresholds fundamentally differ: you could imagine a short expiry threshold in cases where records may be produced to a topic and then subsequently forgotten, or a long expiry where topics are intermittently produced to over the lifetime of the producer.

Therefore, the producer should add should add configuration flag 'metadata.expiry.ms' (default: 5 minutes) to control topic expiry.

Changes will be made to permit a subset of topics to refresh their metadata. In determining which topics' metadata to refresh, the following algorithm will be used:

...