...
- If there are any two log directories with the same UUID, the broker will fail at startup
- If there are any
meta.properties
files missingdirectory.id
, a new UUID is generated, and assigned to that log directory by updating the file - If there are no offline log directories the broker will also create or amend the
directory.ids
field in eachmeta.properties
file as required
If there are offline log directories, the broker
...
- There is a mismatch between the number of entries in
log.dirs
anddirectory.ids
- There is a mismatch between any two
directory.ids
- If the
directory.ids
field is missing in any of themeta.properties
files.
might not be able to determine the UUID for each specific offline log directory, but by diffing diffing directory.ids
with the loaded UUIDs from all directory.id
the set of offline log directory UUIDs can still be determined.
After loading meta.properties
the broker will diff all the UUIDs in directory.id
with the full set of all UUIDs (in directory.ids
) to obtain the set of UUIDs for offline log directories. The sets of both online and offline log directory UUIDs are sent along in the broker registration request to the controller. If log directory that holds the cluster metadata topic is configured separately to a different path — using metadata.log.dir
— then the respective UUID for this log directory is excluded from both online and offline sets, as the broker cannot run if this particular log directory is unavailable.
If a new entry is added in the log.dirs
configuration, the broker can always expand directory.ids
as it can determine the "set of UUIDs for online log directories" + "set of UUIDs for offline log directories" + newly generated UUID for the log directory.
If an entry is removed from log.dirs
the broker can also automatically update directory.ids
as long as no log directories are offline when the broker comes back up. The broker will need to be able to access all meta.properties to determine the new full set of UUIDs. An unresolvable mismatch might occur if some log directory was removed from log.dirs
, and some other log directory is offline. It is not possible to determine which UUID belonged to each of the missing log dirs. The UUID for the removed log directory needs to be removed from directory.ids
but the UUID for the offline log directory should stay. Upon an unresolvable mismatch between the number of entries configured in log.dirs
and found in metada.properties
under directory.ids
the broker will fail at startup
If there are offline log directories, the broker cannot be sure of the the value for directory.ids
should be, as it cannot confirm the directory.id
for any of the offline log directories.
After loading meta.properties
the broker will diff all the UUIDs in directory.id
with the full set of all UUIDs (in directory.ids
) to obtain the set of UUIDs for offline log directories. The sets of both online and offline log directory UUIDs are sent along in the broker registration request to the controller. If log directory that holds the cluster metadata topic is configured separately to a different path — using metadata.log.dir
— then the respective UUID for this log directory is excluded from both online and offline sets, as the broker cannot run if this particular log directory is unavailable.
Replica management
As the broker catches up with metadata, and sees the partitions which it should be hosting, it will check the associated log directory UUID for each partition.
...
- Partition reassignment across directories and across brokers involves different API calls —
AlterPartitionReassignments
andAlterReplicaLogDirs.
Whilst reassigning partitions across brokers into a specific log directory is already possible, it involves an intricate sequence of calls previous calls to AlterReplicaLogDirs and expecting errors as a successful result. Once this work is done we can consolidate these two API calls by extendingAlterPartitionReassignments
to allow target log directories to be specified and deprecateAlterReplicaLogDirs
. This can be done as part of a future KIP. - The only way to know which log directory UUID corresponds to which log directory path is by reading the
meta.properties
files in each broker. A future KIP should expand the DescribeLogDirs RPC response to include log directory UUIDs along with the system path for each log directory. - Partition initialization can be optimized, by having the controller preselect a log directory for new partitions. This would avoid having to wait for the broker to send a
AssignReplicasToDirs
request to indicate the chosen log directory before it is safe for the broker to assume leadership of the partition. Maybe the controller could also take available storage in each log directory into account if the the broker the broker indicates the available storage space for each log directory as part of broker registration. This may be be proposed in a future KIP.
...