Page History

...

• We plan to add a new metric 

  kafka.network:name=ControllerRequestQueueSizeControlPlaneRequestQueueSize,type=RequestChannel

  to show the size of the new controller request queue.
• We will add two new metrics 
  kafka.network:name=ControllerNetworkProcessorIdlePercentControlPlaneNetworkProcessorIdlePercent,type=SocketServer
  kafka.server:name=ControllerRequestHandlerIdlePercentControlPlaneRequestHandlerIdlePercent,type=KafkaRequestHandlerPool
  with the former monitoring the idle percentage of pinned controller network control plane network thread, and the latter monitoring the idle percentage of pinned controller request control plane request handler thread. (Pinned controller threads control plane threads are explained in details below.)
• The meaning of the existing metric

  kafka.network:name=RequestQueueSize,type=RequestChannel

  will be changed, and it will be used to show the size of the data request queue only, which does not include controller requests.
• The meaning of the two existing metrics
  kafka.network:name=NetworkProcessorAvgIdlePercent,type=SocketServer
  kafka.server:name=RequestHandlerAvgIdlePercent,type=KafkaRequestHandlerPool
  will be changed slightly in that they now only cover the threads for data plane threads, not including the pinned threads for controller requests.
• We plan to add a new config for brokers to specify a dedicated listener for controller connections: the "controllercontrol.plane.listener.name" config. A detailed explanation of the config is shown in the "proposed changes" section.
• A new listener-to-endpoint entry dedicated to controller connections need to be added to the "listeners" config and the "advertised.listeners" config.
  Also a new entry needs to be added to the "listener.security.protocol.map" to specify the security protocol of the new endpoint.
  
  

...

Upon detecting a new broker through Zookeeper, the controller will figure out which endpoint it should use to connect to the new broker by first determining the the inter-broker-listener-name. The inter-broker-listener-name is decided by using either the "inter.broker.listener.name" config or the "security.inter.broker.protocol" config. Then by using the "endpoints" section of the broker info, the controller can determine which endpoint to use for a given inter-broker-listener-name. For instance, with the sample json payload listed above, if the controller first determines inter-broker-listener-name to be "INTERNAL", then it knows to use the endpoint "INTERNAL://broker1.example.com:9092" and security protocol PLAINTEXT to connect to the given broker.

Proposed change by using the "

...

control.plane.listener.name" config

Instead of using the inter-broker-listener-name value, we propose to add a new config "controllercontrol.plane.listener.name" for determining the controller endpoints. For instance, if the controller sees that the exposed endpoints by a broker is the following:

{
	"listener_security_protocol_map": {
		"CONTROLLER": "PLAINTEXT"
        "INTERNAL": "PLAINTEXT",
        "EXTERNAL": "SSL"
    },
    "endpoints": [
		"CONTROLLER://broker1.example.com:9091",
        "INTERNAL://broker1.example.com:9092",
        "EXTERNAL://host1.example.com:9093"
    ],
    "host": "host1.example.com",
    "port": 9092,
    "jmx_port": -1,
    "timestamp": "1532467569343",
    "version": 4
}

and the "controllercontrol.plane.listener.name" config is set to value "CONTROLLER", it will use the corresponding endpoint "CONTROLLER://broker1.example.com:9091" and the security protocol "PLAINTEXT" for connections with this broker.

Whenever the "controllercontrol.plane.listener.name" is set, upon broker startup, we will validate its value and make sure it's different from the inter-broker-listener-name value.

If the "controllercontrol.plane.listener.name" config is not set, the controller will fall back to the current behavior and use inter-broker-listener-name value to determine controller-to-broker endpoints.

...

With the dedicated endpoints for controller connections, upon startup a broker will use the "controllercontrol.plane.listener.name" to look up the corresponding endpoint in the listeners list for binding. For instance, in the example given above, the broker will derive the dedicated endpoint to be "CONTROLLER://192.1.1.8:9091". Then it will have a new dedicated acceptor that binds to this endpoint, and listens for controller connections. When a connection is received, the socket will be given to a dedicated processor thread (network thread). The dedicated processor thread reads controller requests from the socket and enqueues them to a new dedicated queue for controller requests, whose default capacity is 20 [2].  On the other side of the controller request queue, a dedicated request handler thread will take requests out, and handles them in the same way as being done today. In summary, we are adding a dedicated acceptor, pinning one processor thread, adding a new request queue, and pinning one request handler thread for controller connections and requests. The two new threads are exclusively for requests from the controller and do not handle data plane requests.

If the "controllercontrol.plane.listener.name" config is not set, then there is no way to tell the dedicated endpoint for controller, and hence there will be no dedicated acceptor, network processor, or request handler threads. The behavior should be exactly same as the current implementation.

...

• Impacts: Controller requests will not longer be blocked by data requests, which should mitigate the effect of stale metadata listed in the motivation section.
• Migration plan: 2 rounds of rolling upgrades are needed to pick up the proposed changes in this KIP. The goal of the first round is to add the controller endpoint, without adding the "controllercontrol.plane.listener.name" config. Specifically, an endpoint with the controller listener name should be added to the "listeners" config, e.g. "CONTROLLER://192.1.1.8:9091"; if the "advertised.listeners" config is explicitly configured and is not getting its value from "listeners", the new endpoint for controller should also be added to "advertised.listeners". After the first round is completed, controller to brokers communication should still behave in the same way that uses the inter-broker-listener-name, since the "controllercontrol.plane.listener.name" is not set yet. In the 2nd round, the "controllercontrol.plane.listener.name" config should be set to the corresponding listener name, e.g. "CONTROLLER". During rolling upgrade of the 2nd round, controller to brokers traffic will start using the "controllercontrol.plane.listener.name", and go through the proposed changes in this KIP.
• No special migration tools are needed.
• The existing behavior will be removed after the PR is merged in.

...