Versions Compared

Old Version 85

changes.mady.by.user Igor Soarez

Saved on

compared with

New Version Current

changes.mady.by.user Igor Soarez

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: authorization for new rpc

...

A AssignReplicasToDirs request including an assignment to Uuid.LOST_DIR conveys that the Broker is wanting to correct a replica assignment into a offline log directory, which cannot be identified.

This request is authorized with CLUSTER_ACTION on CLUSTER.

Proposed changes

Metrics

MBean nameDescription
kafka.server:type=KafkaServer,name=QueuedReplicaToDirAssignments

The number of replicas hosted by the broker that are either missing a log directory assignment in the cluster metadata or are currently found in a different log directory and are queued to be sent to the controller in a AssignReplicasToDirs request.

...

  • As per KIP-866, a separate Controller quorum is setup first, and only then the existing brokers are reconfigured and upgraded.
  • When configured for the migration and while still in ZK mode, brokers will:
    • update meta.properties to generate and include directory.id;
    • send BrokerRegistrationRequest including the log directory UUIDs;
    • shutdown if any directory fails;
    • sends assignments via the  AssignReplicasToDirs RPCnotify the controller of log directory failures via BrokerHeartbeatRequest.
  • During the migration, the controller:
    • persists log directories indicated in broker registration requests in the cluster metadata;
    • relies on heartbeat requests to detect log directory failure instead of monitoring the ZK znode for notifications;
    • still uses full LeaderAndIsr requests to process log directory failures for any brokers still running in ZK modepersists directory assignments received via the AssignReplicasToDirs RPC.
  • The brokers restarting into KRaft mode will want to stay fenced until their log directory assignments for all hosted partitions are persisted in the cluster metadata.
  • The active controller will also ensure that any given broker stays fenced until it learns of all partition to log directory assignments in that specific broker via the new AssignReplicasToDirs RPC.
  • During the migration, existing replicas are assumed and assigned to log directory Uuid.MIGRATING_DIR until the actual log directory is learnt by the active controller from a broker running in KRaft mode.

...