...

Public interfaces

Command line tools

The format  sub-command in the kafka-storage.sh  tool will be updated in the following ways:

...

will produce meta.properties  in a new format described in the following section which includes two new properties directory.id and directory.ids. This sub-command already supports a formatting more than one log directory — by expecting a list of

...

If some or all of the log directories are new, then the format  command should be used. Otherwise update-directories  can be used to update the two properties: directory.id  and directory.ids . This can ensure that each log directory configured in log.dirs  has a unique UUID assigned under the property log.directory  in its meta.properties  and that the full set of UUIDs for all configured log.dirs   is persisted in directory.ids in all meta.properties .—  and "formatting" only the ones that need so. All configured log directories must be available for either format  or update-directoriesfor kafka-storage.sh format to run.

meta.properties

The meta.properties  version field will be bumped from 1 to 2. Two new properties directory.id and directory.ids will be added to the meta.properties file in each log directory, including the metadata.log.dir . The first property, directory.id indicates the UUID for the log directory where the file is located, the second property, directory.ids  lists all the UUIDs for all the configured log directories. If the meta.properties  file doesn't exist for the metadata.log.dir  the Kafka node will fail to start. If the meta.properties  file exists but it doesn't contain these two properties a new one will be generated and the meta.properties  files will be updated. The kafka-storage.sh  tool will be extended to generate and or update the two properties as described in the previous section.

...

Proposed changes

Storage format command

KIP-631 introduced the requirement that storage directories on a node must be formatted. The format subcommand will be updated to ensure each log directory has an assigned UUID and it will persist two new properties in the meta.properties  file:

...

The meta.properties  version field will be bumped from 1 to 2.A new sub-command update-directories  will be introduced to update the two properties for existing and already "formatted" log directoriesto 2.

The UUIDs for each log directory are automatically generated by the tool if there isn't one assigned already in an existing meta.properties  file.

...

• /var/lib/kafka/metadata  has log directory UUID e6umYSUsQyq7jUUzL9iXMQ 
• /mnt/d1  has log directory UUID b4d9ExdORgaQq38CyHwWTA 
• /mnt/d2 has log directory UUID P2aL9r4sSqqyt7bC0uierg 

...

• log directory UUID P2aL9r4sSqqyt7bC0uierg 

If some but not all log directories are unavailable, the broker is able to identify which UUIDs refer to offline log directories.  directories. If there are no new log directories ./bin/kafka-storage.sh update-directories -c /tmp/server.properties would update the three meta.properties  files as necessary to ensure the two properties directory.id  and directory.ids  are present and in sync.

Brokers

Broker lifecycle management

...

If a log directory fails and the active controller learns of the failed log directory at a time when the replica-to-log-directory assignment in the metadata is incorrect then the controller might not know to update the leadership and ISR for some replicas, if some of those replicas are leaders then those partitions will become unavailable. As a fallback safety mechanism in case of any synchronisation issues, when a log directory fails, the broker will check log directory assignments in the metadata and if there are any missing replicas, the broker will use AlterReplicaLogDirs assigning the replicas to the offline log directory, to convey to the controller that these replicas are offline and need leadership and ISR updates. 

...

The cluster needs to be upgraded before configuring multiple entries in log.dirs. As the upgraded brokers come up, the existing meta.properties  files in each broker are updated with a generated directory.id  and directory.ids . After the upgrade, the metadata.version  is feature flag needs to be upgraded using kafka-features.sh. Then the brokers can be reconfigured with multiple entries in log.dirs.

As per KIP-631 storage directories must be formatted with the storage tool, which will generate the new meta.properties files and update the existing one.

Upon being reconfigured with multiple log directories, brokers will register them Upon being reconfigured with multiple log directories, brokers will update and generate meta.properties as necessary to reflect the new log directories. Brokers will then register the log directories with the controller via BrokerRegistration and use AssignReplicasToDirectories to create the partition-logdirectory assignments in the cluster metadata before becoming UNFENCED.

...