THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
Status
Current state: AcceptedUnder Discussion
Discussion thread: https://lists.apache.org/thread/cz9x90883on98k082qd0tskj6yjhox1t
...
We propose adding the following configuration
Name | Description | Default | Valid Values | Priority | Update Mode |
|---|---|---|---|---|---|
| The time in ms that |
a topic partition leader will wait before expiring producer ids. Producer IDs will not expire while a transaction associated to them is still ongoing. Note that producer ids may expire sooner if the last write from the producer id is deleted due to the topic's retention settings. Setting this value the same or higher than |
, but the default should be reasonable for most use cases. | 86400000 (1 day) | [ |
1, …] | low | per-broker |
The following configuration will also be modified:
...
NOTE: Setting producer.id.expiration.ms to be lower than the delivery timeout will increase the likelihood of duplicates. Any expiration of producer IDs means that the producer can no longer take advantage of idempotency guarantees.
This configuration will be dynamic. It can be updated per broker, or for the entire cluster.
Compatibility, Deprecation, and Migration Plan
We can set the default to be -1. This is will be used a flag to signal using the same value as New brokers will start using the new default of 1 day to expire producer IDs. In typical usage, this should not cause any noticeable difference to users since out of sequence records should be very uncommon outside of the retry window. The value can also be configured to match transactional.id.timeout.ms. This will offer backwards compatibility and and continuity of behavior for the use. It will also cover the case where transaction.id.timeout.ms was also using its defaultexpiration.ms if it is necessary to keep the old behavior.
Rejected Alternatives
Have a default value of 7 days
This would be a safe option and match a configuration that did not customize transactional.id.expiration.ms. In the case of a custom configuration for transactional expiration, it would more easily reveal that the new configuration existed. However, since this is still not totally apparent, it is better to default to not changing current behavior.
Have a default
...
value of -1 to use the transaction.id.expiration.ms configured value
Although this would be the most seamless transition for compatibility, it doesn't allow users to see benefits from the new configuration. The default of 1 day should not cause issues with typical clients, so it seems fair to set the default lower.
Make configuration static
Originally this configuration was created to be static. However, in the case of a misbehaving client, it may be necessary to take action to change the value. When the broker is overloading, rolling the cluster may be tricky. By having a dynamic config, we can reduce impact.
The motivation for this change is to have a different (and shorter) period for expiration so that the broker is not easily overwhelmed with all the new idempotent producers. Since this configuration will silently drop idempotency guarantees, it seems safer to make the default more conservative.
Have the flag value as 0 instead of -1
0 would allow us to not have to worry about a valid configuration of 0 ms (immediately expired). However, 0 seems less intuitive as a flag and 0 ms is not all that different in practice to 1, which is a valid value for transactional.id.expiration.ms.