...

For (1), an RPC is generated every time an uncached topic's metadata must be fetched. During periods when a large number of uncached topics are processed (e.g. producer startup), a large number of RPCs may be sent out to the controller in broker(s) in a short period of time. Generally, if there's n unknown topics, then O(n) metadata RPCs will be sent regardless to their proximity in time.

...

In concert, these factors amplify the negative effects of each other, and improvements should be made in order to alleviate topic scalability issues.

Public Interfaces

Adds producer configuration flag metadata.evict.ms (default: 5 minutes) to control topic eviction duration.

...

...

.

...

Proposed Changes

The proposal is to resolve (2), which should reduce the cost of (1) considerably.

...

Therefore, the producer should add configuration flag 'metadata.evicteviction.period.ms' (default: 5 minutes) to control topic eviction.

...

Note in the event of request failures (timeouts), there is no plan to change the current behavior, which is to wait 'retry.backoff.ms' before retrying.

Public Interfaces

Adds producer configuration flag metadata.eviction.period.ms (default: 5 minutes) to control topic eviction duration.

/** <code>metadata.eviction.period.ms</code> */
public static final String METADATA_EVICTION_PERIOD_MS_CONFIG = "metadata.eviction.period.ms";
private static final String METADATA_EVICTION_PERIOD_MS_DOC =
        "Controls how long the producer will cache metadata for a topic that's not being accessed. " +
        "If the elapsed time since a topic was last produce to exceeds the metadata eviction duration, " +
        "then the topic's metadata is forgotten and the next access to it will force a metadata " +
        "fetch request.";

...
    .define(METADATA_EVICTION_PERIOD_MS_CONFIG,
            Type.LONG,
            2 * 60 * 1000,
            atLeast(5000),
            Importance.LOW,
            METADATA_EVICTION_PERIOD_MS_DOC)

Compatibility, Deprecation, and Migration Plan

...