...
In concert, these factors amplify the negative effects of each other, and improvements should be made in order to alleviate topic scalability issues.
Public Interfaces
...
Adds producer configuration flag metadata.expiry.ms (default: 5 minutes)
to control topic expiry duration.
Code Block | ||
---|---|---|
| ||
/** <code>metadata.expiry.ms</code> */
public static final String METADATA_EXPIRY_MS_CONFIG = "metadata.expiry.ms";
private static final String METADATA_EXPIRY_MS_DOC =
"Controls how long the producer will cache metadata for a topic that's not being accessed. " +
"If the elapsed time since a topic was last produce to exceeds the metadata expiry duration, " +
"then the topic's metadata is forgotten and the next access to it will force a metadata " +
"fetch request.";
...
.define(METADATA_EXPIRY_MS_CONFIG,
Type.LONG,
5 * 60 * 1000,
atLeast(1000),
Importance.MEDIUM,
METADATA_EXPIRY_MS_DOC) |
Proposed Changes
The proposal is to resolve (2), which should reduce the cost of (1) considerably.
The producer has two values of interest: an expiry threshold for topic metadata, which is used to remove an unused topic from the working set at a future time (currently hard-coded to 5 minutes), and a metadata refresh threshold, which is used to periodically refresh topic metadata (defined by metadata.max.age.ms
). While seemingly similar, these two thresholds fundamentally differ: you could imagine a short expiry threshold in cases where records may be produced to a topic and then subsequently forgotten, or a long expiry where topics are intermittently produced to over the lifetime of the producer.
Therefore, the producer should add should add configuration flag 'metadata.expiry.ms
' (default: 5 minutes) to control topic expiry.
Changes will be made to permit a subset of topics to refresh their metadata. In determining which topics' metadata to refresh, the following algorithm will be used:
...