You are viewing an old version of this page. View the current version.

Compare with Current View Page History

Version 1 Next »

Status

Current state: Under Discussion

Discussion thread: not yet started

JIRA: https://issues.apache.org/jira/projects/KAFKA/issues/KAFKA-16081

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

For Kafka, an SSL connection occupies approximately 100KB of memory, while a plaintext connection occupies around 250 bytes, resulting in a memory footprint ratio of approximately 400:1. Therefore, there should be a limitation for SSL connections in broker wide to prevent potential OOM situations caused by an excessive number of SSL connections.

Currently, we have max.connections configuration at both the broker and listener levels, which allows us to limit the maximum number of active connections on each listener and the overall broker. However, the current implementation poses a challenge in scenarios where we have one broker which supports 10000 SSL connections, along with two SSL listerner. Configuring the connection count accurately becomes an issue in this design. For example, if we set the connection count to 5,000 for each listener, we cannot precisely control the SSL connection count to be 10,000, especially if one listener is heavily used while the other has fewer connections.

Public Interfaces

No new interfaces or will be added.

Two new metrics will be added to track the total numer of ssl connections and all connections , and it will be a yammer Meter.

  • kafka.network:type=ConnectionQuotas,name=SslConnectionsCount
  • kafka.network:type=ConnectionQuotas,name=ConnectionsCount

A new broker configuration max.ssl.connections option will be added to limit the total number of  ssl connections.

This is in addition to the existing max.connections.per.ip and max.connections config that will continue to limit the total number of connections and from each host ip address. When this limitation is reached, new SSL connections will be restricted while non-SSL connections will not be affected until one or more SSL connections are disconnected. This will be a dynamic broker-wide config that can be updated without restarting the broker.

Config option: Name: max.ssl.connections Type: Int Default value: Int.MaxValue

Proposed Changes

Acceptor will stop accepting new connections when the broker's max.connections limit is reached. New connections will be accepted as soon as a connection is closed by any of the Processors. Acceptor will also stop accepting new connections when its listener's listener.name.{listener}.max.connections limit is reached. New connections will be accepted as soon as a connection is closed by any of the Processors of that listener. Inter-broker connections will be protected in multi-listener brokers by closing client connections to accommodate inter-broker connections. Any time spent by Acceptor waiting for connections to close will also be included in the new AcceptorBlockedPercent metric. The existing max.connections.per.ip config will be applied without any changes. Connections dropped due to hitting the per-ip limit will not appear in the AcceptorBlockedPercent metric since these connections are accepted and then dropped.

The behavior when the SSL connection limit is reached will be similar to reaching the maximum connection limit at the broker level. It is important to note that  this configuration will not affect non-SSL listeners.



Compatibility, Deprecation, and Migration Plan

  • What impact (if any) will there be on existing users?
  • If we are changing behavior how will we phase out the older behavior?
  • If we need special migration tools, describe them here.
  • When will we remove the existing behavior?

Test Plan

Describe in few sentences how the KIP will be tested. We are mostly interested in system tests (since unit-tests are specific to implementation details). How will we know that the implementation works as expected? How will we know nothing broke?

Rejected Alternatives

If there are alternative ways of accomplishing the same thing, what were they? The purpose of this section is to motivate why the design is the way it is and not some other way.

  • No labels