...
Current state: Under Discussion
Discussion thread: here [Change the link from the KIP proposal email archive to your own email thread]
JIRA: here [Change the link from KAFKA-1 to your own ticket] https://lists.apache.org/thread/8dqvfhzcyy87zyy12837pxx9lgsdhvft
JIRA:
Jira | ||||||
---|---|---|---|---|---|---|
|
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
...
Storage will have to be formatted — using the storage format tool — not only when a log directory is added, but also when one is removed from configuration.
Metadata records
meta.properties
Two new properties storage.id
and storage.ids
will be added to the meta.properties
file in each log directory. The first property, storage.id
indicates the UUID for the log directory where the file is located, the second property, storage.ids
lists all the UUIDs for all the configured log directories. If the meta.properties
file doesn't exist for the cluster metadata partition the Kafka node will fail to start. If the meta.properties file exist but it doesn't contain these two properties a new one will be generated and the meta.properties
files will be re-written. The kafka-storage CLI tool will be extended to generate and write the two properties when the format command is used.
Metadata records
RegisterBrokerRecord
will and RegisterBrokerRecord
will and BrokerRegistrationChangeRecord
both have two new fields:
...
The storage format tool must be run when new log directories are added or removed from a broker’s configuration. This command will :
...
ensure each log directory has an
...
assigned UUID and it will persist two new properties in the meta.properties file:
- A property named
storage.id
indicting the UUID for the log directory where the meta.properties file is located - A property named
storage.ids
indicating the complete list of all UUIDs for each configured log directory
...
- .
Having a persisted UUID as a file at the root of each log directory allows the broker to identify the log directory regardless of the mount path.
Having a persisted list of all UUIDs for all configured log directories allows the broker to determine the UUIDs of unavailable (offline) log directories, as the UUID file is also meta.properties files for the offline log directories are likely to be unavailable.
...
When the broker starts up and initializes the LogManager, it will load the UUID for each log directory (storage.id
) and the list of all log directory UUIDs (storage.ids
), by reading a new metadata file named id
the meta.properties file at the root of the log directory. If different meta.properties files indicate different storage.ids
the broker will fail at startup. It will also load the directories
file at the root of the broker metadata log directory, and diff diff the UUIDs with the list of all UUIDs to obtain the list of UUIDs for offline log directories. The lists of both online and offline log directory UUIDs are sent along in the broker registration request to the controller.
...