Status

Current state: "Voting"

Vote thread: here

Discussion thread: here

JIRA: https://issues.apache.org/jira/browse/KAFKA-18775

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

Currently, when using MetadataQuorumCommand to add a controller, users must provide a controller.properties configuration file. This file is required for the command to retrieve the metadata local path and endpoints needed to add voters. However, this approach has several limitations:

1. Limited Accessibility: The node executing the tool must have direct access to the metadata path of the node being added or removed. This restricts the ability to use node A to manage node B, as node A may not have access to the metadata folder on node B.

2. Dependency on Node Configuration: The tool requires access to the configuration of the node being managed.

However, the essential information for these operations – the "directory UUID" and the "endpoints" – can be obtained through active controller in-memory state and ClusterImage.

By leveraging these, we can simplify the process of adding/removing voters, allowing the command to be executed without direct access to the target node's metadata directory.

Public Interfaces

CLI

The kafka-metadata-quorum.sh tool introduces a new option —-controller-id for the add-controller subcommand.

Adding a controller

For adding a controller, only --controller-id is needed option.

bin/kafka-metadata-quorum.sh --bootstrap-server localhost:9092 add-controller --controller-id <id>

bin/kafka-metadata-quorum.sh --bootstrap-controller localhost:9093 add-controller --controller-id <id>

Removing a controller

For removing a controller, the —-controller_directory_id option is no longer required.

bin/kafka-metadata-quorum.sh --bootstrap-server localhost:9092 remove-controller --controller-id <id>

bin/kafka-metadata-quorum.sh --bootstrap-controller localhost:9093 remove-controller --controller-id <id>

Public APIs

Admin.java

/**
 * Add a new voter node to the KRaft metadata quorum.
 * Note that this is a convenient method for simple deployment, and not idempotent. 
 * For a complicated scenario, e.g., Node Disk Failure, there might have  
 * different directory uuid with same node id, in this scenario, please  
 * go with {@link #addRaftVoter(int, Uuid, Set)}.
 *
 * @param voterId           The node ID of the voter.
 */
default AddRaftVoterResult addRaftVoter(int voterId) {  
    return addRaftVoter(voterId, UUID.zero_uuid, null, new AddRaftVoterOptions());
}

/**
 * Remove a voter node from the KRaft metadata quorum.
 *
 * @param voterId           The node ID of the voter.
 */
default RemoveRaftVoterResult removeRaftVoter(int voterId) {
    return removeRaftVoter(voterId, UUID.zero_uuid, new RemoveRaftVoterOptions());
}

RPC Changes

AddRaftVoterRequest.json

diff --git a/clients/src/main/resources/common/message/AddRaftVoterRequest.json b/clients/src/main/resources/common/message/AddRaftVoterRequest.json
index 74b7638ea2..27a6e5face 100644
--- a/clients/src/main/resources/common/message/AddRaftVoterRequest.json
+++ b/clients/src/main/resources/common/message/AddRaftVoterRequest.json
@@ -18,7 +18,7 @@
   "type": "request",
   "listeners": ["controller", "broker"],
   "name": "AddRaftVoterRequest",
-  "validVersions": "0",
+  "validVersions": "0-2",
   "flexibleVersions": "0+",
   "fields": [
     { "name": "ClusterId", "type": "string", "versions": "0+", "nullableVersions": "0+",
@@ -27,9 +27,9 @@
       "about": "The maximum time to wait for the request to complete before returning."},
     { "name": "VoterId", "type": "int32", "versions": "0+",
       "about": "The replica id of the voter getting added to the topic partition." },
-    { "name": "VoterDirectoryId", "type": "uuid", "versions": "0+",
+    { "name": "VoterDirectoryId", "type": "uuid", "versions": "0+", "nullableVersions": "1+",
       "about": "The directory id of the voter getting added to the topic partition." },
-    { "name": "Listeners", "type": "[]Listener", "versions": "0+",
+    { "name": "Listeners", "type": "[]Listener", "versions": "0+", "nullableVersions": "1+",
       "about": "The endpoints that can be used to communicate with the voter.", "fields": [
       { "name": "Name", "type": "string", "versions": "0+", "mapKey": true,
         "about": "The name of the endpoint." },

RemoveRaftVoterRequest.json

diff --git a/clients/src/main/resources/common/message/RemoveRaftVoterRequest.json b/clients/src/main/resources/common/message/RemoveRaftVoterRequest.json
index 7d11086e53..2181ecd9ff 100644
--- a/clients/src/main/resources/common/message/RemoveRaftVoterRequest.json
+++ b/clients/src/main/resources/common/message/RemoveRaftVoterRequest.json
@@ -18,14 +18,14 @@
   "type": "request",
   "listeners": ["controller", "broker"],
   "name": "RemoveRaftVoterRequest",
-  "validVersions": "0",
+  "validVersions": "0-1",
   "flexibleVersions": "0+",
   "fields": [
     { "name": "ClusterId", "type": "string", "versions": "0+", "nullableVersions": "0+",
       "about": "The cluster id of the request."},
     { "name": "VoterId", "type": "int32", "versions": "0+",
       "about": "The replica id of the voter getting removed from the topic partition." },
-    { "name": "VoterDirectoryId", "type": "uuid", "versions": "0+",
+    { "name": "VoterDirectoryId", "type": "uuid", "versions": "0+", "nullableVersions": "1+",
       "about": "The directory id of the voter getting removed from the topic partition." }
   ]
 }

Proposed Changes

Make both controller Directory UUID and Endpoints optional

We’ll handle both fields on the server side since the information already exists:

• Directory ID: derive from in-memory LeaderState.

• Controller endpoints: derive from the ClusterImage.

Because the ClusterImage may lag behind, using it for endpoints is not strictly idempotent. To reflect that, Admin.java introduces two convenience methods for add/remove controller that include prominent warnings in javadoc so users understand the risk and only use them when appropriate.

Upon server-side changes live in KafkaRaftClient.java:

• handleAddVoterRequest

• handleRemoveVoterRequest

Error handling

• During add raft voter, detecting multiple observers sharing the same node id — the request is rejected with a clear error IllegalStateException indicating the duplicate node id and instructing the user to resolve the conflict before retrying.

MetadataQuorumCommand add-controller changes

Add a new option —-controller-id to add-controller subcommand.

        addControllerParser
            .addArgument("--controller-id", "-i")
            .help("The id of the controller to add. This option should be used with bootstrap controller.")
            .type(Integer.class)
            .action(Arguments.store());

• If —-controller-id is provided, invoke new method Admin#addRaftVoter(int)

• If —-command-config and —-controller-id are both provided, the config file provided by --command-config will only be applied in Admin client initialization.

  • the description for --command-config will be changed to "Property file containing configs to be passed to Admin Client. For add-controller, the file is used to specify the controller properties as well unless --controller-id is provided."

• If neither --command-config nor controller-id is provided, an exception will be thrown:

  • throw new TerseException("You must use --command-config or --controller-id option to add a controller.");

MetadataQuorumCommand remove-controller changes

diff --git a/tools/src/main/java/org/apache/kafka/tools/MetadataQuorumCommand.java b/tools/src/main/java/org/apache/kafka/tools/MetadataQuorumCommand.java
index dba7951aa4..f3bdbbeffa 100644
--- a/tools/src/main/java/org/apache/kafka/tools/MetadataQuorumCommand.java
+++ b/tools/src/main/java/org/apache/kafka/tools/MetadataQuorumCommand.java
@@ -471,7 +471,6 @@ public class MetadataQuorumCommand {
         removeControllerParser
             .addArgument("--controller-directory-id", "-d")
             .help("The directory ID of the controller to remove.")
-            .required(true)
             .action(Arguments.store());

• The —-controller-directory-id is no longer required, we can leverage on the new method Admin#removeRaftVoter(int)

• If —-controller-directory-id is explicitly provided, invoke Admin#removeRaftVoter(int, Uuid) 

Compatibility, Deprecation, and Migration Plan

This change should be backward compatible:

• The —-command-config option remains available in add-controller.

• The --controller-directory-id option in remove-controller is now optional but still supported.

This KIP introduces new methods in Admin.java and introduce nullable fields for 2 RPCs with no breaking change. 

Test Plan

New test cases will be added to MetadataQuorumCommandTest.java to validate:

• Adding a controller with --controller-id.

• Removing a controller without explicitly providing --controller-directory-id.

Will also add new integration tests for the two new methods in Admin.java.

Rejected Alternatives

• Deprecate —-command-config option in add-controller and --controller-directory-id option in remove-controller.

  The main reason not to deprecate these two parameters is that they were only just introduced in 4.0, so deprecating them in a 4.x release feels a bit too soon. Also, the --command-config can be used in a different user scenario, where the user can still provide the configuration file to add-controller if they already have it locally.

• Use admin APIs to get directory UUID and controller endpoints, this brings extra network communication overhead.

  1. The Admin#describeMetadataQuorum method can provide the directory UUID.

  2. The Admin#describeConfigs method, utilizing the bootstrap.controller address, can be used to retrieve the necessary endpoints.