...

Because this KIP is introducing a new metadata record alongside a new MetadataVersion, it means that existing clusters who have a stale controller registration will not be able to unregister it, and unblock feature upgrades thereafter.  

One approach is for the controller to allow unregistering controllers non-durably if the metadata version does not support the UnregisterControllerRecord. This gives operators a path to migrate existing clusters onto a MV that supports this record, where they can persist the unregister record. One issue with this approach is that the UX is not good, requiring the user to unregister the controller twice.

Another approach is to allow the active controller to unregister observer controllers that the user specifies as part of a MV upgrade to support UnregisterControllerRecord. The main issue with this approach is that it makes the UX for kafka-features upgrade pretty complex. Additionally, this approach may be overkill in non-managed deployments, where operators can simply provision a controller to update its stale registrationThe main reason for not supporting this in existing clusters is that in many environments, operators can simply bring up another controller node to "refresh" its registration. Additionally, the interest of keeping this design simple, some of the potential workarounds for existing clusters have been moved to the Rejected Alternatives section.

Test Plan

Add an integration test for unregistering a controller in both static quorum and dynamic quorum clusters. 

...

This approach would be one way existing clusters could support unregistering stale controller registrations without updating the MV. However, the main issue with this approach is that it is unsafe. A user who unregisters a controller before updating the software versions on all controllers to support this feature would crash the controllers with an older software version. 

Non-durably unregister controllers

This would be a workaround for existing clusters with a stale registration, where the active controller's state machine could unregister a controller, but the on-disk data does not change. The main reason against implementing this would be a confusing UX during scenarios with node restarts. Additionally, it is confusing for operators to run the unregister CLI command again after updating the MV to support the UnregisterControllerRecord.

Allow the active controller to unregister observer controllers as part of a MV upgrade

Another approach to clear stale registrations from existing clusters is to allow the active controller to unregister observer controllers that the user specifies as part of a MV upgrade to support UnregisterControllerRecord. The main issue with this approach is that it makes the UX for kafka-features upgrade very complex. Additionally, this approach may be overkill in non-managed deployments, where operators can simply provision a controller to update its stale registration.