THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
Initially, we will not support upgrading a cluster from legacy mode to KIP-500 mode. This is in keeping with the experimental nature of KIP-500 mode. A future KIP will describe and implement an upgrade process from legacy mode to KIP-500 mode.
Before being used in KIP-500 mode, the storage directories on a node must be formatted. This requirement prevents system administrators from accidentally enabling KIP-500 mode by simply making a configuration change. Requiring formatting also prevents mistakes, since Kafka no longer has to guess if an empty storage directory is a newly directory or one where a system error prevented any data from showing up.
Nodes
Currently, a ZooKeeper cluster must be deployed when running Kafka. This KIP will eliminate that requirement, as well as the requirement to configure the addresses of the zookeeper nodes on each broker.
...
| Code Block |
|---|
INITIAL
sent initial heartbeart, |
got back an epoch |
V
ACTIVE -------------------------> SHUTDOWN
lease expired or revoked | ^ ^
(stale metadata, | | lease restored | controller granted
id conflict, | | with new broker epoch | controlled shutdown
unable to communicate, V | |
etc.) FENCED -----------------------------+ |
Public Interfaces
Command-Line Tools
There will be a new command-line tool, kafka-storage.sh.
| Code Block |
|---|
kafka-storage.sh: inspect and modify kafka storage directories
subcommands:
info: print out information about kafka storage directories
-c|--config [path]: the configuration file to use
format: format the kafka storage directories
-f|--force: format even if the directories are not empty.
-d|--directory [path]: format only a single directory, rather than all directories |
The info subcommand will display information about each directory, such as whether it exists, is formatted, etc.
The format subcommand will initialize the subdirectories.
A cluster configured in KIP-500 mode will contain the following line in the meta.properties file in each directory:
| Code Block |
|---|
kip.500.mode=enabled |
If the storage directories in a node have not been properly formatted, the node will not be able to start up in KIP-500 mode. On the other hand, formatting will continue to be optional for legacy mode.
Configurations
| Configuration Name | Possible Values | Description |
|---|---|---|
| process.roles | null broker controller broker,controller | If this is null (absent) then we are in legacy mode. Otherwise, we are in KIP-500 mode and this configuration determines what roles this process should play: broker, controller, or both. |
| controller.listeners | null a listener name | A comma-separated list of the names of the listeners used by the KIP-500 controller. This is required if this process is a KIP-500 controller. The legacy controller will not use this configuration Despite the similar name, note that this is different from the "control plane listener" introduced by KIP-291. The "control plane listener" is used on brokers, not on controllers. |
| controller.connect | null a comma-separated list of controller URIs. | A comma-separated list of controller URIs that KIP-500 brokers should connect to on startup. Required for nodes running in KIP-500 mode. |
| controller.id | a 32-bit ID | The controller id for this server. If this is not specified, we will default to the broker.id plus 3000. |
| registration.heartbeat.interval.ms | 1000 | The length of time between broker heartbeats. |
| registration.lease.timeout.ms | 8000 | The length of time that a broker lease lasts if no heartbeats are made. |
...