Page History

...

No new interfaces or will be added. The size of the blocking queue used for new connections will be set to 20 50 for each Processor. This matches the backlog queue size for incoming connections on the server socket. Kafka brokers use the Java default backlog size of 50 and this limit is not configurable. Like polling interval in SocketServer, it is unlikely that users will require this number the queue size to be changed. During normal operation, a small limit allows progress to be made on new channels as well as existing channels. Since Selector is woken up when new connections arrive or existing connections are ready to be processed, this limit does not introduce any unnecessary delays in connection processing.

...

The connection queue for Processors will be changed to ArrayBlockingQueue with a fixed size of 20 50. Acceptor will use round-robin allocation to allocate each new connection to the next available Processor to which the connection can be added without blocking. If a Processor's queue is full, the next Processor will be chosen. If the connection queue on all Processors are full, Acceptor blocks until the connection can be added to the selected Processor. No new connections will be accepted during this period. The amount of time Acceptor is blocked can be monitored using the new AcceptorIdlePercent metric.

...

In typical scenarios, Kafka uses long-lived connections, so a small queue size (20 per-processor is proposed in this KIP) is sufficient to ensure that new connections are processed promptly and existing connections are not left behind. Queue size of 50 per-processor is proposed in this KIP to match the server socket backlog queue size for which we use the Java default. The goal of this KIP is to protect the broker in scenarios when a very large number of clients connect at the same time. This is likely to be true only for short bursts and hence the small queue size of 20 50 should be sufficient to ensure fairness in channel processing while protecting the broker from the surge. It is not clear that the number will need to be tweaked for different deployments since queue size is per-processor and the number of processors can be configured using num.network.threads.

...