Page History

...

Current state: Under Discussion

Discussion thread: here [Change the link from the KIP proposal email archive to your own email thread]
JIRA: here [Change the link from KAFKA-1 to your own ticket] https://lists.apache.org/thread/8dqvfhzcyy87zyy12837pxx9lgsdhvft 

JIRA:

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

...

Storage will have to be formatted — using the storage format tool — not only when a log directory is added, but also when one is removed from configuration.

Metadata records

meta.properties

Two new properties storage.id and storage.ids will be added to the meta.properties file in each log directory. The first property, storage.id indicates the UUID for the log directory where the file is located, the second property, storage.ids  lists all the UUIDs for all the configured log directories. If the meta.properties  file doesn't exist for the cluster metadata partition the Kafka node will fail to start. If the meta.properties file exist but it doesn't contain these two properties a new one will be generated and the meta.properties  files will be re-written. The kafka-storage CLI tool will be extended to generate and write the two properties when the format command is used.

Metadata records

RegisterBrokerRecord will and RegisterBrokerRecord will and BrokerRegistrationChangeRecord both have two new fields:

...

The storage format tool must be run when new log directories are added or removed from a broker’s configuration. This command will :

...

ensure each log directory has an

...

assigned UUID and it will persist two new properties in the meta.properties file:

• A property named storage.id indicting the UUID for the log directory where the meta.properties file is located
• A property named storage.ids indicating the complete list of all UUIDs for each configured log directory

...

• .

Having a persisted UUID as a file at the root of each log directory allows the broker to identify the log directory regardless of the mount path.
Having a persisted list of all UUIDs for all configured log directories allows the broker to determine the UUIDs of unavailable (offline) log directories, as the UUID file is also meta.properties files for the offline log directories are likely to be unavailable.

...

When the broker starts up and initializes the LogManager, it will load the UUID for each log directory (storage.id ) and the list of all log directory UUIDs (storage.ids), by reading a new metadata file named idthe meta.properties file at the root of the log directory. If different meta.properties files indicate different storage.ids the broker will fail at startup. It will also load the directories file at the root of the broker metadata log directory, and diff diff the UUIDs with the list of all UUIDs to obtain the list of UUIDs for offline log directories. The lists of both online and offline log directory UUIDs are sent along in the broker registration request to the controller.

...