...

This KIP aims to solve both problems by introducing a new RPC and a new method of generating Producer ID blocks using the metadata log as storage.

Public Interfaces

New AllocateProducerId RPC AllocateProducerIds RPC to be used by brokers to request a new block of IDs from the controller

AllocateProducerIdRequestAllocateProducerIdsRequest => BrokerId BrokerEpoch
  BrokerId => int32
  BrokerEpoch => int64

AllocateProducerIdResponseAllocateProducerIdsResponse => ErrorCode ProducerIdStart ProducerIdLen
  ErrorCode => int16
  ProducerIdStart => int64
  ProducerIdLen => int32 

...

An authorization error will be considered fatal and should cause the broker to terminate. This indicates the broker is incorrectly configured to communicate with the controller. All other error types should be treated as transient and the broker should retry.

The AllocateProducerId AllocateProducerIds RPC should be rate limited using the existing throttling mechanism. This will help guard against malicious or malfunctioning clients.

...

The broker will now defer to the controller for allocating blocks of PIDs. This is done by sending an AllocateProducerIdRequest AllocateProducerIdsRequest to the controller using an inter-broker channel. If the request fails, the broker will retry for certain transient errors. The broker should use the ID block returned by the RPC (as opposed to waiting to replicate the ProducerIdRecord metadata record).

...