Versions Compared

Old Version 71

changes.mady.by.user Colin McCabe

Saved on

compared with

New Version 72

changes.mady.by.user Colin McCabe

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Code Block
 *                           +-------------+
 *                           | NOT_RUNNING |
 *                           +-------------+
 *                                  |
 *                                  ↓
 *                           +-------------+
 *                           |   STARTING  |
 *                           +-------------+
 *                                |      |
 *                                ↓      |
 * +----------------------------------+  |
 * | RECOVERING_FROM_UNCLEAN_SHUTDOWN |  |
 * +----------------------------------+  |
 *                                |      |
 *                                ↓      ↓
 *                           +-------------+
 *                           | REGISTERING |
 *                           +-------------+
 *                                  |
 *                                  ↓
 *                           +-------------+
 *                           |    FENCED   |
 *                           +-------------+
 *                                |      ↑
 *                                ↓      |
 *                           +-------------+   +-----------------------------+
 *                           |    RUNNING  |--→| PENDING_CONTROLLED_SHUTDOWN |
 *                           +-------------+   +-----------------------------+
 *                                                           |
 *                                                           ↓
 *                                                     +---------------+
 *                                                     | SHUTTING_DOWN |
 *                                                     +---------------+
 *                                                           |
 *                                                           ↓
 *                                                      +-------------+
 *                                                      | NOT_RUNNING |
 *                                                      +-------------+

Public Interfaces

Command-Line Tools

There will be a new command-line tool, kafka-storage.sh.

meta.properties

When a storage directory is in use by a cluster running in kip-500 mode, it will have a new version of the meta.properties file.  Since the current version is 0, the new version will be 1.  For now, the contents of the file will be the same in version 1 as version 0.

Here is an example of a version 1 meta.properties file:

Code Block
$ cat /tmp/kafka-logs/meta.properties
#
#Tue Dec 01 10:08:08 PST 2020
cluster.id=51380268-1036-410d-a8fc-fb3b55f48033
version=1
broker.id=0

Command-Line Tools

There will be a new command-line tool, kafka-storage.sh.

Code Block
$ ./bin/kafka-storage.sh -h
usage: kafka-storage [-h] {info,format,random-uuid} ...

The Kafka storage tool.

positional arguments:
  {info,format,random-uuid}
    info                 Get information about the Kafka log directories on this node.
    format
Code Block
kafka-storage.sh: inspect and modify kafka storage directories
subcommands:
    info: print out information about kafka storage directories
        -c|--config [path]: the configuration file to use
    format: format the kafka storage directories
        -c|--config [path]:      Format the configuration file to useKafka log directories on this node.
    random-uuid    -f|--force: format even if the directories are not empty.
      Print a random UUID.

optional arguments:
  -h, --help        -d|--directory [path]: format only a singleshow directory,this ratherhelp thanmessage alland directories

The info subcommand will display information about each directory, such as whether it exists, is formatted, etc.

The format subcommand will initialize the subdirectories.

A cluster configured in KIP-500 mode will contain the following line in the meta.properties file in each directory:

Code Block
kip.500.mode=enabled

...

exit

kafka-storage.sh will have three subcommands: info, format, and random-uuid.

info

Code Block
$ ./bin/kafka-storage.sh info -h
usage: kafka-storage info [-h] --config CONFIG

optional arguments:
  -h, --help             show this help message and exit
  --config CONFIG, -c CONFIG
                         The Kafka configuration file to use.

The info command will give information about the configured storage directories.  Example output:

Code Block
Found log directory:
  /tmp/kafka-logs

Found metadata: MetaProperties(clusterId=51380268-1036-410d-a8fc-fb3b55f48033)

format

Code Block
$ ./bin/kafka-storage.sh format -h
usage: kafka-storage format [-h] --config CONFIG --cluster-id CLUSTER_ID [--ignore-formatted]

optional arguments:
  -h, --help             show this help message and exit
  --config CONFIG, -c CONFIG
                         The Kafka configuration file to use.
  --cluster-id CLUSTER_ID, -t CLUSTER_ID
                         The cluster ID to use.
  --ignore-formatted, -g

When running kip-500 mode, the storage directories must be formatted using this command prior to starting up the brokers and controllers.

If any of the storage directories are formatted, the command will normally fail.  This behavior can be changed by passing the --ignore-formatted option.  When this option is passed, the format command will skip over already formatted directories rather than failing.

random-uuid

Code Block
$ ./bin/kafka-storage.sh random-uuid -h
usage: kafka-storage random-uuid [-h]

optional arguments:
  -h, --help             show this help message and exit

The random-uuid command prints out a random UUID to stdout.

Code Block
$ ./bin/kafka-storage.sh random-uuid
51380268-1036-410d-a8fc-fb3b55f48033

Configurations

Configuration NamePossible ValuesDescription
process.roles

null

broker

controller

broker,controller

If this is null (absent) then we are in legacy mode.

Otherwise, we are in KIP-500 mode and this configuration determines what roles this process should play: broker, controller, or both.

controller.listener.names

If non-null, this must be a comma-separated list of listener names.

When communicating with the controller quorum, the broker will always use the first listener in this list.

A comma-separated list of the names of the listeners used by the KIP-500 controller. This is required if this process is a KIP-500 controller. The legacy controller will not use this configuration

Despite the similar name, note that this is different from the "control plane listener" introduced by KIP-291.  The "control plane listener" is used on brokers, not on controllers.

listeners

A comma-separated list of the configured listeners.  For example,

INTERNAL://192.1.1.8:9092, EXTERNAL://10.1.1.5:9093, CONTROLLER://192.1.1.8:9094

This configuration is now required.
controller.connect

If non-null, this must be a comma-separated list of all the controller voters, in the format:

{controller-id}@{controller-host):{controller-port}

When in KIP-500 mode, each node must have this configuration, in order to find out how to communicate with the controller quorum.

Note that this replaces the "quorum.voters" config described in KIP-595.

This configuration is required.

controller.id

a 32-bit ID

The controller id for this server.  Only required if this server is a controller.

broker.ida 32-bit IDThe broker id for this server.  Only required if this server is a broker.
registration.heartbeat.interval.ms2000The length of time between broker heartbeats.
registration.lease.timeout.ms18000The length of time that a broker lease lasts if no heartbeats are made.
metadata.log.dirIf set, this must be a path to a log directory.This configuration determines where we put the metadata log.  if it is not set, the metadata log is placed in the first log directory from log.dirs.
controller.quorum.fetch.timeout.msMaximum time without a successful fetch from the current leader before a new election is started.New name for quorum.fetch.timeout.ms

controller.quorum.election.timeout.ms

Maximum time without collected a majority of votes during the candidate state before a new election is retriedNew name for quorum.election.timeout.ms
controller.quorum.election.backoff.max.msMaximum exponential backoff time (based on the number if retries) after an election timeout, before a new election is triggered.New name for quorum.election.backoff.max.ms
controller.quorum.request.timeout.msMaximum time before a pending request is considered failed and the connection is droppedNew name for quorum.request.timeout.ms
controller.quorum.retry.backoff.msInitial delay between request retries. This config and the one below is used for retriable request errors or lost connectivity and are different from the election.backoff configs aboveNew name for quorum.retry.backoff.ms
controller.quorum.retry.backoff.max.msMax delay between requests. Backoff will increase exponentially beginning from quorum.retry.backoff.msNew name for quorum.retry.backoff.max.ms

...