...

Creating a new connection adds CPU overhead to the broker. KIP-306 mitigated the issue of connection storm due to un-authorized connections (e.g., misconfigured clients). However, connections storms may also come from (mostly) well-behaved clients. One example is when deploying a new application may cause temporary connection storm due to a large number of clients starting up and creating connections to the cluster at the same time. Another example is clients that create a new connection for each produce/consume request, causing high connection rate to the brokers. A very high connection creation rate may stop broker from doing other useful work, causing high request latencies or even URPs.

To address this issue, the KIP proposes to add the ability to set a limit on the rate with which the broker accepts new connections. To ensure that some clients do not take over most of the connection creation rate quota, the KIP also adds the ability to set a limit on connection creation rate per IP. 

Public Interfaces

Broker configurations for broker-wide and per-listener connection rate limits

A new broker configuration option will be added to limit the total rate at which non-inter-broker connections will be accepted on the broker. Connections on the inter-broker listener will be permitted even if the configured broker-wide limit is reached. This will be a dynamic config that can be updated without restarting the broker. 

...

The config may be prefixed with a listener prefix to specify a different listener-specific limit: listener.name.{listenerName}.max.connection.creation.rate. Listener-specific limits will be applied in addition to the broker-wide limit. If a listener-specific limit is not specified, each listener can create connections with the rate up to the broker-wide limit as long as the total rate is also within the broker-wide limit. If a broker has multiple listeners, connections on the inter-broker listener will always succeed as long as connection creation rate is within that listener's rate limit. The behavior of the proposed broker-wide and per-listener configs is consistent with max.connections broker configuration (KIP-402).

A new broker configuration option will be added to limit the rate at which connections will be accepted for each IP address. New connections for the IP will be dropped once the limit is reached. This will also be a dynamic config that can be updated without restarting the broker:

...

Dynamic configurations for a new entity type ips

Per-IP connection creation rate quota will be configured as a dynamic quota for entity type "ips".

• Default quota for <ip-address> will be stored in Zookeeper at /config/ips/<default>
• Quota for a specific IP address for which quota override is defined will be stored in Zookeeper at /config/ips/<ip-address>

If per-IP quota is not configured, the default value is Int.MaxValue to ensure backward compatibility.

Tools

kafka-configs.sh will be extended to support per-IP connection rate quotas.  A new entity type “ips” will be added with the following key (and a value of type Int):

• connection_creation_rate : The total connection rate limit for the connections from a specific IP address. 

For example:

bin/kafka-configs  --bootstrap-server localhost:9091 --alter --add-config 'connection_creation_rate=100' --entity-name 93.284.53.13 --entity-type ips

Default connection rate quotas for an IP address can be configured by omitting entity name. For example:

bin/kafka-configs  --bootstrap-server localhost:9091 --alter --add-config 'connection_creation_rate=100' --entity-type ips

Connection rate throttling behavior

The connection creation rate limits will be applied to the same quota window configuration (quota.window.size.seconds with 1 second default) as existing produce/fetch quotas and request rate quota (KIP-124). Since limit on connection creation rate on the broker is also a type of quota, this approach will keep it consistent with the existing quota implementations on the broker. If connection creation rate on the broker exceeds the broker-wide limit, the broker will delay accepting a new connection by an amount of time that brings the rate within the limit. If a listener-specific limit is specified, and the connection rate on that listener exceeds the limit, the broker will delay accepting new connection on that listener by an amount of time that brokers the rate of the listener within the listener limit. The maximum delay applied will be the quota window size (which also means that the minimum connection rate limit is effectively 1 connection creation / second). 

If connection creation rate is reached for a specific IP address, the connection will be dropped. The broker will continue dropping connections for that IP until the rate for the IP is within the per-IP connection creation rate limit.

Metrics

No new metrics will be added. The existing metric (kafka.network:type=Acceptor,name=AcceptorBlockedPercent,listener={listenerName}) that tracks the amount of time Acceptor is blocked from accepting connections will now additionally include the amount of time Acceptor is blocked due to hitting connection create limit (in addition to the time blocked due to hitting the maximum limit on currently active connections). 

...