Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Code Block
$ ./bin/kafka-cluster.sh decommission -h
usage: kafka-cluster decommission [-b,-c,-h,-i]

optional arguments:
  -b, --bootstrap-server a list of host/port pairs to use for establishing the connection to the kafka cluster.
  -c, --config           a property file containing configs to be passed to Admin Client.
  -f, --force            true if we should force the decommissioning even if this broker is the only replica for some partitions.
  -h, --help             show this help message and exit
  -i, --id               the ID of the broker to decommission

The decommission command removes the registration of a specific broker ID.  Note that this will also remove the broker ID from all replica assignmentsIt will use make a DecommissionBrokerRequest in order to do this.

Configurations

Configuration NamePossible ValuesNotes
process.roles

null

broker

controller

broker,controller

If this is null (absent) then we are in legacy mode.

Otherwise, we are in KIP-500 mode and this configuration determines what roles this process should play: broker, controller, or both.

controller.listener.names

If non-null, this must be a comma-separated list of listener names.

When communicating with the controller quorum, the broker will always use the first listener in this list.

A comma-separated list of the names of the listeners used by the KIP-500 controller. This configuration is required if this process is a KIP-500 controller. The legacy controller will not use this configuration

Despite the similar name, note that this is different from the "control plane listener" introduced by KIP-291.

listeners

A comma-separated list of the configured listeners.  For example,

INTERNAL://192.1.1.8:9092, EXTERNAL://10.1.1.5:9093, CONTROLLER://192.1.1.8:9094

This configuration is now required.
controller.connect

If non-null, this must be a comma-separated list of all the controller voters, in the format:

{controller-id}@{controller-host):{controller-port}

When in KIP-500 mode, each node must have this configuration, in order to find out how to communicate with the controller quorum.

Note that this replaces the "quorum.voters" config described in KIP-595.

This configuration is required for both brokers and controllers.

controller.id

a 32-bit ID

The controller id for this server.  Only required if this server is a controller.

broker.ida 32-bit IDThe broker id for this server.  Only required if this server is a broker.
initial.broker.registration.timeout.ms60000When initially registering with the controller quorum, the number of milliseconds to wait before declaring failure and exiting the broker process.
broker.heartbeat.interval.ms3000The length of time between broker heartbeats.
broker.registration.timeout.ms18000The length of time that a broker lease lasts if no heartbeats are made.
metadata.log.dirIf set, this must be a path to a log directory.This configuration determines where we put the metadata log.  if it is not set, the metadata log is placed in the first log directory from log.dirs.
controller.quorum.fetch.timeout.msMaximum time without a successful fetch from the current leader before a new election is started.New name for quorum.fetch.timeout.ms

controller.quorum.election.timeout.ms

Maximum time without collected a majority of votes during the candidate state before a new election is retriedNew name for quorum.election.timeout.ms
controller.quorum.election.backoff.max.msMaximum exponential backoff time (based on the number if retries) after an election timeout, before a new election is triggered.New name for quorum.election.backoff.max.ms
controller.quorum.request.timeout.msMaximum time before a pending request is considered failed and the connection is droppedNew name for quorum.request.timeout.ms
controller.quorum.retry.backoff.msInitial delay between request retries. This config and the one below is used for retriable request errors or lost connectivity and are different from the election.backoff configs aboveNew name for quorum.retry.backoff.ms
controller.quorum.retry.backoff.max.msMax delay between requests. Backoff will increase exponentially beginning from quorum.retry.backoff.msNew name for quorum.retry.backoff.max.ms

...

There will be a new error code, INVALID_CLUSTER_ID, that the controller will return if the broker tries to register with the wrong cluster ID.

BROKER_REQUIRED_FOR_PARTITIONS

...

AdminClient

There will be a new AdminClient RPC, decommissionBroker.

Code Block
languagejs
DecommissionBrokerResult decommissionBroker(int brokerId, DecommissionBrokerOptions options)

public class DecommissionBrokerResult {
    public KafkaFuture<Void> all();
}

public class DecommissionBrokerOptions extends AbstractOptions<DecommissionBrokerOptions> {
}

RPCs

Obsoleting the Metadata Propagation RPCs

...

Code Block
languagejs
{
  "apiKey": 59,
  "type": "request",
  "name": "DecommissionBrokerRequest",
  "validVersions": "0",
  "flexibleVersions": "0+",
  "fields": [
    { "name": "BrokerId", "type": "int32", "versions": "0+",
      "about": "The broker ID to decommission." },
    { "name": "Force", "type": "bool", "versions": "0+", "default": "false",
      "about": "True if we should force the decommissioning even if this broker is the only replica for some partitions." }
  ]
}   

{
  "apiKey": 59,
  "type": "response",
  "name": "DecommissionBrokerResponse",
  "validVersions": "0",
  "flexibleVersions": "0+",
  "fields": [
    { "name": "ThrottleTimeMs", "type": "int32", "versions": "0+",
      "about": "Duration in milliseconds for which the request was throttled due to a quota violation, or zero if the request did not violate any quota." }
    { "name": "ErrorCode", "type": "int16", "versions": "0+",
      "about": "The error code, or 0 if there was no error." },
  ]
}

...

  • NONE if the decommissioning succeeded
  • NOT_CONTROLLER if the node that the request was sent to is not the controller
  • UNSUPPORTED_VERSION if KIP-500 mode is not enabled
  • BROKER_NOT_AVAILABLE if the given broker ID is not registered.BROKER_REQUIRED_FOR_PARTITIONS if the given broker is the last replica for at least one partition, and force was not specified.

Record Formats

RegisterBrokerRecord

...