...
A new broker configuration option will be added to limit the total number of connections that may be active on the broker at any time. This is in addition to the existing max.connections.per.ip
config that will continue to limit the number of connections from each host ip address. When the limit is reached, new connections will not be accepted until one or more existing connections are closed. The limit This will be a dynamic listener broker-wide config that can be updated without restarting the broker.
Config option: Name: max.connections
Type:
Default value: Int
Int.MaxValue
The config may be prefixed with listener prefix to specify different listener-specific limits for different listeners, enabling inter-broker connections to be created even if there are a large number of client connections on a different listener. Config option: Name: max.connections
Type:
Default value: Int
Int.MaxValue
Listener-specific limits will be applied in addition to the broker-wide limit. If a listener-specific limit is not specified, each listener can create up to the broker-wide limit as long as the total is within the limit. If a broker has multiple listeners, connections on the inter-broker listener will always succeed as long as they are within that listener's limit. In this case, the least-recently used connection on another listener will be closed to accommodate the inter-broker connection.
Proposed Changes
Acceptor accepts new connections and allocates them to Processors
using round-robin allocation. In the current implementation, Acceptor
accepts as fast as possible and adds new connections to unbounded queues associated with each Processor.
...
Acceptor
will stop accepting new connections when the broker's max.connections
limit is reached. New connections will be accepted as soon as a connection is closed by any of the Processors
. Acceptor
will also stop accepting new connections when its listener's listener.name.{listener}.max.connections
limit is reached. New connections will be accepted as soon as a connection is closed by any of the Processors
of that listener. Inter-broker connections will be protected in multi-listener brokers by closing client connections to accommodate inter-broker connections. Any time spent by Acceptor
waiting for connections to close will also be included in the new AcceptorIdlePercent
metric. The existing max.connections.per.ip
config will also be applied without any changes. Connections dropped due to hitting the per-ip limit will not appear in the AcceptorIdlePercent
metric since these connections are accepted and then dropped.
...
In typical scenarios, Kafka uses long-lived connections, so a small queue size is sufficient to ensure that new connections are processed promptly and existing connections are not left behind. Queue size of 20 per-processor is proposed in this KIP to ensure that the server socket backlog queue size for which we use the Java default of 50 can be processed by 3 network threads without blocking. The goal of this KIP is to protect the broker in scenarios when a very large number of clients connect at the same time. This is likely to be true only for short bursts and hence the small queue size of 20 should be sufficient to ensure fairness in channel processing while protecting the broker from the surge. It is not clear that the number will need to be tweaked for different deployments since queue size is per-processor and the number of processors can be configured using num.network.threads.