Page History

...

The motivations here are similar to KIP-854 Separate configuration for producer ID expiry.  Idempotent producers became the default in Kafka since KIP-679: Producer will enable the strongest delivery guarantee by default as a result of this all producer instances will be assigned PID. The increase of number of PIDs stored in Kafka brokers by ProducerStateManager exposes the broker to OOM errors if it has a high number of producersof producers, rogue or misconfigured client(s). Here are few of use-cases that might cause increase of PIDs: 

• Clients fail after produce and enter a repeated restart state for a while. 
• Users initialise a new Producer instance every time they encounter any type of error instead of waiting for the producer instance to auto recover.
• Users who initialise new Producer per batch. 

As a result of this the broker will hit OOM and become offline. The only way to recover from this is to increase the heap.  

...