Page History

...

The motivations here are similar to KIP-854 Separate configuration for producer ID expiry.  Idempotent producers became the default in Kafka since KIP-679: Producer will enable the strongest delivery guarantee by default unless otherwise specified at the client side as  as a result of this all producer instances will be assigned to PIDsPID. The increase of number of PIDs stored in Kafka brokers by ProducerStateManager expose exposes the broker to OOM errors if it has a high number of producersof producers, rogue or misconfigured client(s). As a result of this the broker will hit OOM and become offline. The only way to recover from this is to increase the heap.  

KIP-854 added separated config to expire PID from transaction IDs however the broker is still exposed to OOM if it has high number of PID before producer.id.expiration.ms  is exceeded. And decreasing the value of producer.id.expiration.ms  will impact all clients which not desired all the time. It would be more beneficial to better to target only inefficient users and stopping them from crowding the map of PIDs to their ProducerState  by ProducerStateManager.

This KIP propose proposes to throttle the number PIDs at the leader of the partition by adding a new rating quota that will be applied during handling the PRODUCE request. This way the broker can reject only risky users early on in the process and protect itself without impacting good behaving users.

...