Page History

...

Producer ids are used by idempotent and transaction producers.  The brokers keep a small amount of metadata (e.g. producer id, epoch, sequence number, etc.) in memory for every partition that the idempotent producer produced to.   This metadata is maintained on every replica and it's recovered from logs and snapshots even if brokers restart.  The producer id and its metadata is removed after it's been inactive for a certain time controlled by the `transactional.id.timeout.ms` configuration setting, the default is 7 days.  The KIP-98 - Exactly Once Delivery and Transactional Messaging has details on producer ids and related protocols and data structures.

In idempotent producers, a new producer id is created when KafkaProducer is created.  A badly written application may frequently create new KafkaProducer objects.  This is not optimal in general, but specifically for idempotent producers, doing so would pollute broker memory with producer ids and related metadata.  Even though the metadata for each producer id is small, creating too many producer ids could run brokers out of memory.

The ProducerIdCount metric reflects the total count of producer ids maintained at each broker, and can be used to set up alerts so that this the abovementioned pattern can proactively detected and action could be taken before too many producer ids run the broker out of memory.

...