Status

Current state: Under DiscussionAccepted

Discussion thread: https://lists.apache.org/thread/kc87jkgyvf9x7nwmgwrhx6fs6w0tqymj

Vote thread: https://lists.apache.org/thread/rdpjmmqdxzog2m555r2wrncfn40zjf54

JIRA:

...

However, there is currently no way to unregister a controller, like there is for brokers via UnregisterBrokerRequest and UnregisterBrokerRecord. This means stale controller registrations can block feature upgrades. This KIP proposes adding support for operators to manually unregister controllers like they can with brokers.

The main One important use cases for unregistering a controller are for dynamic quorum clusters case for this KIP is to remove controller registrations from KRaft observers (i.e. clusters with KIP-853 enabled), since controller processes can now become observers of the KRaft log (i.e. they replicate the log but do not nodes that replicate the log but do not participate in leader election or committing data) and persist a controller registration to from the metadata log. However, there is a static quorum edge case where modifications to controller.quorum.voters can allow for a stale ControllerRegistrationRecord  to end up in the log. In either quorum mode, this KIP allows for operators to remove those controller registrations from the metadata log.This means operators can remove these stale registrations to unblock feature upgrades on their cluster.

Public Interfaces

New RPC 

...

{
  "apiKey": 93,
  "type": "request",
  "listeners": ["broker", "controller"],
  "name": "UnregisterControllerRequest",
  "validVersions": "0",
  "flexibleVersions": "0+",
  "fields": [
    { "name": "ControllerId", "type": "int32", "versions": "0+",
      "about": "The controller ID to unregister." }
  ]
}

UnregisterControllerResponse

This request can return the following errors:

• an UNSUPPORTED_VERSION  error if the cluster's MetadataVersion does not support UnregisterControllerRecord 
• a CONTROLLER_ID_NOT_REGISTERED  error if no registration exists for the requested controller ID. This is similar to the BROKER_ID_NOT_REGISTERED error in the case of brokers.
• a NOT_CONTROLLER  error if the request does not arrive at the active controller
• an INVALID_REQUEST error if the request arrives at the active controller. This is known to be a "mistaken" request, as it is expected controllers are not running when they are unregistered. See the User Experience for more details.

UnregisterControllerResponse

{
  "apiKey

{
  "apiKey": 93,
  "type": "response",
  "name": "UnregisterControllerResponse",
  "validVersions": "0",
  "flexibleVersions": "0+",
  "fields": [
    { "name": "ThrottleTimeMs", "type": "int32", "versions": "0+",
      "about": "Duration in milliseconds for which the request was throttled due to a quota violation, or zero if the request did not violate any quota." },
    { "name": "ErrorCode", "type": "int16", "versions": "0+",
      "about": "The error code, or 0 if there was no error." },
    { "name": "ErrorMessage", "type": "string", "versions": "0+", "nullableVersions": "0+",
      "about": "The top-level error message, or `null` if there was no top-level error." }
  ]
}

...

UnregisterControllerRecord

{{
  "apiKey": 29,
  "type": "metadata",
  "name": "UnregisterControllerRecord",
  "apiKeyvalidVersions": 29"0",
  "typeflexibleVersions": "metadata0+",
  "fields": [
    { "name": "UnregisterControllerRecordControllerId",
  "type": "int32", "validVersionsversions": "0+",
      "flexibleVersionsabout": "0+",
  "fields": [
    { "name": "ControllerId", "type": "int32", "versions": "0+",
      "about": "The controller id." }
  ]
}

This KIP will also introduce a new MetadataVersion to support this new metadata record.

CLI changes

kafka-cluster 

The controller id." }
  ]
}

This KIP will also introduce a new MetadataVersion to support this new metadata record.

CLI changes

kafka-cluster 

Add a unregister-controller  for manually unregistering controllers. This command would be similar to how the unregister command works for brokers. Below is an example invocation.

kafka-cluster unregister-controller --controller-id 1

When the user executes this command to unregister controller 1:

1. UnregisterControllerRequest is sent to the active controller
2. The active controller writes an UnregisterControllerRecord to the metadata log
3. When this record is committed, return a response to the user for unregistering the controller
4. The active controller's state machine removes the registration for controller 1, meaning feature upgrades no longer consider node 1's supported features
5. The registration from controller 1 is removed from the metadata image

kafka-cluster unregister-controller  is a command for users when they want to unregister a controller from the cluster. This command should only be run after a controller is stopped, and the operator does not intend to bring it back. kafka-cluster unregister-controller  works irrespective of the quorum mode. 

kafka-metadata-quorum 

Add the --unregister flag to the kafka-metadata-quorum remove-controller command. When this flag is set, invoking this command with --unregister set will remove the controller as a KRaft voter and unregister itAdd a unregister-controller  for manually unregistering controllers. This command would be similar to how the unregister command works for brokers. Below is an example invocation.

kafka-metadata-clusterquorum unregisterremove-controller --controller-id 99901 --controller-directory-id EXAMPLE_UUID --unregister

When the user executes this command to unregister controller 9990:

1. UnregisterControllerRequest is sent to the active controller
2. The active controller writes an UnregisterControllerRecord to the metadata log
3. When this record is committed, return a response to the user for unregistering the controller
4. The active controller's state machine removes the registration for controller 9990, meaning feature upgrades no longer consider node 9990's supported features
5. The registration from controller 9990 is removed from the metadata image

kafka-cluster unregister-controller  is a command for users when they want to unregister a controller from the cluster. This command should only be run after a controller is stopped, and the operator does not intend to bring it back. kafka-cluster unregister-controller  works irrespective of the quorum mode. 

This command supports unregistering any controller registration in any quorum mode, such as:

• Any voter in a dynamic quorum
• Any observer in a dynamic quorum
• Any stale "voter" in a static quorum as a result of a misconfiguration of controller.quorum.voters 

In any of these cases, it is expected that the controller process has stopped before the operator unregisters it via these tools. If the controller process has not stopped, it will attempt to re-register with the cluster.

kafka-metadata-quorum 

Add the --unregister flag to the kafka-metadata-quorum remove-controller command. When this flag is set, invoking this command with --unregister set will remove the controller as a KRaft voter and unregister it. Below is an example invocation.

kafka-metadata-quorum remove-controller --controller-id 9990 --controller-directory-id EXAMPLE_UUID --unregister

When the user executes this command, kafka tries to remove 9990 as a voter AND unregister it:

1. RemoveRaftVoterRequest  is sent to the active controller
2. The KRaft leader writes a VotersRecord without voter 9990 to the metadata log
3. When this record is committed, return a response to the user for removing the voter
4. UnregisterControllerRequest is sent to the active controller if steps 1-3 were successful
   1. If steps 1-3 were not successful, return the error and direct the user to use kafka-cluster unregister-controller instead.
5. The active controller writes an UnregisterControllerRecord to the metadata log
6. When this record is committed, return a response to the user for unregistering the controller
7. The active controller's state machine removes the registration for controller 9990, meaning feature upgrades no longer consider node 9990's supported features
8. The registration from controller 9990 is removed from the metadata image

...

this command, kafka tries to remove 1 as a voter AND unregister it:

1. RemoveRaftVoterRequest  is sent to the active controller
2. The KRaft leader writes a VotersRecord without voter 1 to the metadata log
3. When this record is committed, return a response to the user for removing the voter
4. UnregisterControllerRequest is sent to the active controller if steps 1-3 were successful
   1. If steps 1-3 were not successful, return the error and direct the user to use kafka-cluster unregister-controller instead.
5. The active controller writes an UnregisterControllerRecord to the metadata log
6. When this record is committed, return a response to the user for unregistering the controller
7. The active controller's state machine removes the registration for controller 1, meaning feature upgrades no longer consider node 1's supported features
8. The registration from controller 1 is removed from the metadata image

The main use case for this command is to remove a node from the voter set in a dynamic KRaft quorum, AND unregister it from the cluster all within the same CLI command. Since this is a common usage pattern, running this command with --unregister can be thought of as a "built-in" script that provides a smooth UX for decommissioning voters in a dynamic quorum. Running the command with --unregister will fail when the cluster does not support dynamic quorum to be consistent with the behavior of the command when --unregister is not set.

User Experience

The main use cases of these CLI tools are listed below. When trying to unregister a controller, it is assumed that the operator has stopped a node before unregistering it and does not intend to bring that node back in the near future. This is because after unregistering a node, the active controller no longer checks its supported feature levels when validating a feature upgrade. 

Remove and unregister a KRaft voter in a dynamic quorum

1. Stop the voter
2. Run kafka-metadata-quorum remove-controller with the --unregister flag

Remove a KRaft voter in a dynamic quorum and keep it registered as an observer controller

1. Run kafka-metadata-quorum remove-controller without the --unregister flag

Unregister an observer controller in a static or dynamic quorum

1. Stop the observer
2. Run kafka-cluster unregister-controller

Unregister a voter in a static KRaft quorum when the static voter set is mistakenly configured

1. Stop the voter who was mistakenly put in controller.quorum.voters 
2. Run kafka-cluster unregister-controller
3. Ensure the stopped voter is not part of controller.quorum.voters on every Kafka node

Proposed Changes

Controller Changes

...

Because this KIP is introducing a new metadata record alongside a new MetadataVersion, it means that existing clusters who have a stale controller registration will not be able to unregister it, and unblock feature upgrades thereafter. The main reason for not supporting this in existing clusters is that in many environments, operators can simply bring up another controller node with the same node ID to "refresh" its registration. Additionally, the interest of keeping this design simple, some of the potential workarounds for existing clusters have been moved to the Rejected Alternatives section.

...

This approach would be one way existing clusters could support unregistering stale controller registrations without updating the MV. However, the main issue with this approach is that it is unsafe. A user who unregisters a controller before updating the software versions on all controllers to support this feature would crash the controllers with an older software version.

Additionally, combined mode deployments where the broker and controller use the same ID make reusing the same metadata record for unregistering both brokers and controller too complex compared to introducing a new record for unregistering controllers.

Non-durably unregister controllers

...