...

Public Interfaces

Broker Configuration

• Config name: max.idempotence.batches.to.retain

• Config type: Int

• Default value: 5

• Constraint: at least 5

Introduce a new config on the broker, as the broker must know how much memory to allocate. Operators can set a limitation on the broker side to prevent malicious producers. This configuration only takes effect for idempotent/transactional producers. Prior to this KIP, idempotent/transactional producers .

...

are limited to a maximum of 5 for max.in.flight.requests.per.connection. Setting a value greater than 5 on these older producers causes a ConfigException at initialization. To ensure backward compatibility, the new broker configuration max.idempotence.batches.to.retain

...

Config type: Int

...

Default value: 5

must have a minimum constraint of 5. 

This KIP preserves the existing default of 5 for both max.idempotence.batches.to.retain and max.in.flight.requests.per.connection. While higher defaults could improve throughput, such a change is not the goal of this KIP. We defer this discussion to a future proposal.

...

ProduceRequest / ProduceResopnse

...

Initial State (Safety First): When a connection is first established, the produce sends at most min(5, max.in.flight.requests.per.connection) for each node. This ensures safety with any broker version. Old brokers hardcode NUM_BATCHES_TO_RETAIN to 5. To ensure compatibility, the producer initially limits in-flight requests to min(5, max.in.flight.requests.per.connection) until it discovers the broker's actual capability via the ProduceResponse. This prevents sending excessive in-flight requests to older brokers that cannot handle them.

Discovery: Upon receiving the first ProduceResponse from a partition leader, the producer checks for the presence of the MaxIdempotenceBatchesToRetain tagged field.

...