Page History

...

Currently, Replicas are considered offline if the hosting broker is offline. Additionally, replicas will also be considered offline if the replica references a log directory UUID (in the new field partitionRecord.Assignment.Directory) that is not present in the hosting Broker's latest registration under OnlineLogDirs and either:

• the log directory UUID is UUID.OfflineDir
• the hosting broker's latest registration indicates multiple online log directories. i.e. brokerRegistration.OnlineLogDirs.length > 1
• the hosting broker's latest registration indicates that there are offline directories. i.e. brokerRegistration.OfflineLogDirs == true

...

• If there are no indicated online log directory UUIDs the request is invalid and the controller replies with an error 42 – INVALID_REQUEST.

• If multiple log directories are registered the broker will remain fenced until the controller learns of all the partition to log directory placements in that broker - i.e. no remaining replicas assigned to Uuid.UnknownDir . The broker will indicate these using the AssignReplicasToDirs RPC.

  • The broker remains fenced by not wanting to unfence itself in heartbeat requests until the number of mismatching replica to log directory assignments is zero. This number is represented by the new metric QueuedReplicaToDirAssignments.
• If multiple log directories are registered and some of them are new (not present in previous registration) then these log directories are assumed to be empty. If they are not, the broker will use the AssignReplicasToDirs RPC to correct assignment and choose not to become UNFENCED before the metadata is correct.
• In the special case where previous broker registration indicates a single online log directory and no offline log directories, and the inbound broker registration request indicates more than one log directory, and one of the indicated log directories is the same one previously registered, then a logical update to all partitions in that broker takes place, assigning the replica's directory to the single directory previously registered – i.e. it is assumed that all replicas are still in the same directory, and this transition to JBOD avoids creating partition change records. This same logic is considered in every node while consuming and caching metadata changes. Any metadata snapshot created after this change explicitly refers the exact log directory UUID for each partition in that broker in each respective PartitionRecord.
• If the registration request does not indicate any offline directories (i.e. OfflineLogDirs=false) and it does not include all directories previously registered (i.e. OnlineLogDirs in the previous registration contains UUIDs that are not present in the request's OnlineLogDirs) then the Controller assumes that those directories have been removed from configuration and that any hosted partitions in those directories will need to be re-created by the broker in the remaining configured log directories. So another logical update takes place here, applied to all partitions assigned to the removed directory UUIDs, assigning them instead to UUID.OfflineDir.

Brokers whose registration indicates that multiple log directories are configured remain FENCED until all log directory assignments for that broker are learnt by the active controller and persisted into metadata.

...