Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Status

Current state:  AcceptedUnder Discussion

Discussion thread: https://lists.apache.org/thread/cz9x90883on98k082qd0tskj6yjhox1t

...

We propose adding the following configuration

Name

Description

Default

Valid Values

PriorityUpdate Mode

producer.id.expiration.ms

The time in ms that

the ProducerStateManager

a topic partition leader will wait before expiring producer ids. Producer IDs will not expire while a transaction associated to them is still ongoing. Note that producer ids may expire sooner if the last write from the producer id is deleted due to the topic's retention settings. Setting this value the same or higher than delivery.timeout.ms can help prevent expiration during retries and protect against message duplication, but the default should be reasonable for most use cases.

86400000 (1 day)

[1, …]

lowper-broker

The following configuration will also be modified:

...

NOTE: Setting producer.id.expiration.ms to be lower than the delivery timeout will increase the likelihood of duplicates. Any expiration of producer IDs means that the producer can no longer take advantage of idempotency guarantees.

This configuration will be dynamic. It can be updated per broker, or for the entire cluster.

Compatibility, Deprecation, and Migration Plan

...

Although this would be the most seamless transition for compatibility, it doesn't allow users to see benefits from the new configuration. The default of 1 day should not cause issues with typical clients, so it seems fair to set the default lower.

Make configuration static

Originally this configuration was created to be static. However, in the case of a misbehaving client, it may be necessary to take action to change the value. When the broker is overloading, rolling the cluster may be tricky. By having a dynamic config, we can reduce impact.