Table of Contents

Status

Current state: Under Discussion

Discussion thread: not Not yet started

JIRA: https://issues.apache.org/jira/projects/KAFKA/issues/KAFKA-16081

...

Currently, we have max.connections configuration at both the broker and listener levels, which allows us to limit the maximum number of active connections on each listener and the overall broker. However, the current implementation poses presents a challenge in scenarios where how to effectively controlling the number of SSL connections. For example, if we have one broker which supports 10000 a broker supporting 10,000 SSL connections , along with two SSL listerner. Configuring the connection count accurately listeners, how to configure limit of each listener becomes an issue in this design. For example, if we set the connection count to 5,000 for each listener, we cannot precisely control the total SSL connection connections count to be 10,000, especially if one listener is heavily used while the other has fewer connections.

The new configuration could work together with max.connections to control ssl and non-ssl connections. For examplem, set max.connections to 10000, max.ssl.connections to 5000, which means non-ssl connection radio is set at 5000.

Public Interfaces

No new interfaces or will be added.

Two new metrics will be added to track the total numer of ssl connections and all connections , and it will be a yammer Meter.

• kafka.network:type=ConnectionQuotas,name=SslConnectionsCount
• kafka.network:type=ConnectionQuotas,name=ConnectionsCount

A new broker configuration max.ssl.connections option will be added to limit the total number of  ssl connections.

This is in addition to the existing max.connections.per.ip and max.connectionsconfig that will continue to limit the total number of connections and from each host ip address. When this limitation is reached, new SSL connections will be restricted while non-SSL connections will not be affected until one or more SSL connections are disconnected. This will be a dynamic broker-wide config that can be updated without restarting the broker.

Config option: Name: max.ssl.connections Type: Int Default value: Int.MaxValue

Proposed Changes

Acceptor will stop accepting new connections when the broker's max.connections limit is reached. New connections will be accepted as soon as a connection is closed by any of the Processors. Acceptor will also stop accepting new connections when its listener's listener.name.{listener}.max.connections limit is reached. New connections will be accepted as soon as a connection is closed by any of the Processors of that listener. Inter-broker connections will be protected in multi-listener brokers by closing client connections to accommodate inter-broker connections. Any time spent by Acceptor waiting for connections to close will also be included in the new AcceptorBlockedPercent metric. The existing max.connections.per.ip config will be applied without any changes. Connections dropped due to hitting the per-ip limit will not appear in the AcceptorBlockedPercent metric since these connections are accepted and then dropped.

The behavior when the SSL connection limit is reached will be similar to reaching the maximum connection limit at the broker level. It is important to note that  that this configuration will not affect non-SSL listeners. 

Based on the current implementation, we will use the LRU algorithm to close connections for the listener in the following two scenarios:

1. Total number of connections exceeds the broker's connection limit.
2. The number of connections exceeds the listener's connection limit.

I propose adding an additional condition: if the number of SSL connections exceeds the SSL connection limit and the current listener is using SSL protocol.

Compatibility, Deprecation, and Migration Plan

• What impact (if any) will there be on existing users?
• If we are changing behavior how will we phase out the older behavior?
• If we need special migration tools, describe them here.
When will we remove the existing behavior?

No externally visible interface changes are proposed in this KIP. During normal operations, this is unlikely to result in any impact. When a large number of ssl connections are made to the broker at the same time, connections may be established slower, and the least recently used connections may be disconnected

Test Plan

Describe in few sentences how the KIP will be tested. We are mostly interested in system tests (since unit-tests are specific to implementation details). How will we know that the implementation works as expected? How will we know nothing broke?

...