Current state: Under Discussion
Discussion thread: here
JIRA:
Draft PR: https://github.com/apache/kafka/pull/15888
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
While tuning Producer ID expiration on Produce State Manager, there is a mismatch on how the configurations are exposed: while producer.id.expiration.ms is exposed as dynamic public configuration, producer.id.expiration.check.interval.ms is static and internal.
This does not allow to adjust this values when needed: e.g. if there are too many producer ids, an operator may want to reduce the expiration to remove the producer ids faster; but without checking the expiration faster there's no benefit and the producer ids will pill up anyway.
This has lead to issues like
Broker configuration producer.id.expiration.check.interval.ms to be: public (i.e. not internal) and dynamic.
Config:
name:
producer.id.expiration.check.interval.ms |
doc:
The interval at which to remove producer IDs that have expired due to <code>producer.id.expiration.ms</code> passing. |
In practice, this means re-scheduling the Producer Expiration check thread in UnifiedLog component.
No compatibility issues. Config already exists, and will just be documented and available as dynamic config.
Extend UnifiedLogTest to validate re-scheduling of Producer Expiration check thread.
If there are alternative ways of accomplishing the same thing, what were they? The purpose of this section is to motivate why the design is the way it is and not some other way.