This page is meant as a template for writing a KIP. To create a KIP choose Tools->Copy on this page and modify with your content and replace the heading with the next KIP number and a description of your issue. Replace anything in italics with your own description.

Status

Current state: Accepted

Discussion thread: here

JIRA: here

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

It's pretty common to have several listeners defined on Kafka brokers. Each listener has its own pool of network threads. In many cases some listeners handle a lot less traffic than others and typically don't need the same number of threads. This KIP proposes optionally setting the sizes of these pool individually using per listener configurations. This will allow fine tuning the number of threads to dynamically accommodate traffic spikes or slightly reduce memory usage when using listeners with different usages.

Public Interfaces

The existing configuration "num.network.threads" will be updated to also support being set on specific listener via the "listener.name.<NAME>" notation. Like this other per-listener configuration, the listener name must be provided in lower case.

For example, a valid configuration could be:

server.properties
listeners=EXTERNAL://:9092,INTERNAL://:9093,CONTROLLER://:9094
advertised.listeners=EXTERNAL://localhost:9092,INTERNAL://localhost:9093,CONTROLLER://localhost:9094
listener.security.protocol.map=EXTERNAL:PLAINTEXT,INTERNAL:PLAINTEXT,CONTROLLER:PLAINTEXT
num.network.threads=16
listener.name.internal.num.network.threads=2

With this configuration, we would end up with:

  • 16 threads in the External listener network thread pool
  • 2 threads in the Internal listener network thread pool

The control plane listener is not touched by this proposal, it always has a single thread.

Like with the existing configuration, "num.network.threads" per listener configurations will also be reconfigurable. For example, a user could run the following command to have 3 threads in the Internal listener network thread pool.

./bin/kafka-configs.sh --bootstrap-server localhost:9092 --entity-type brokers --entity-default --alter --add-config listener.name.internal.num.network.threads=3

The documentation will be adjusted to mention it can optionally be configured on a specific listener:
Listener-level limits may also be configured by prefixing the config name with the listener prefix, for example, listener.name.internal.num.network.threads

Proposed Changes

Most of the changes are in SocketServer where the logic creating Acceptors and Processors lives. To make this reconfigurable, the data plane acceptor will have to implement ListenerReconfigurable. Also the logic to create Processors will be moved directly inside Acceptor instead of being driven by SocketServer.

Compatibility, Deprecation, and Migration Plan

This is a new configuration option. This changes nothing for existing Kafka environments.

Rejected Alternatives

None

  • No labels