Status

Current state: Under DiscussionAccepted

Discussion thread: https://lists.apache.org/thread/ko478l71jf9hqhhg328tcdr46nj2wcz9

Vote thread: https://lists.apache.org/thread/1bhgq12dpc4s20pfos88t1wgz3l4y7lk

JIRA:

...

This KIP provides a simple solution to an availability issue exposed by the auto-join feature proposed in KIP-853 and how AddRaftVoterRequest  RPC is currently handled. The problem is because the active controller does not send a response to complete the AddRaftVoterRequest until after the new voter set is committed, and that KRaft (and Kafka in general) only support one in-flight request to a node. Consider the following scenario:

Some controller A that is automatically joining by sending the AddRaftVoterRequest  RPC is the same controller whose Fetch is needed to commit the new voter set. A clear example of this is when bootstrapping with --standalone and having controllers auto-join, as the first controller to auto-join will increase the voter set size from 1 to 2. The active controller needs controller A to complete a Fetch  RPC to complete the AddRaftVoterRequest  RPC, but controller A cannot send a FetchRequest  until its AddRaftVoterRequest  returns or times out. However, this case also applies when The general case being described here is going from a voter set of size X to size X + 1, where a minority of X + 1 nodes from the new voter set are unavailable.

The reason this scenario causes unavailability is as follows: the current in-flight request for controller A is the AddRaftVoterRequest  RPC, which cannot complete until after controller A first replicates the new voter set, and sends another fetch to the leader. This state will expire the leader's checkQuorumTimer and cause it to resign, since the majority of nodes (which includes controller A) will not be able to fetch in time. Thus, the current implementation of AddRaftVoterRequest  will cause an unnecessary leadership failover and election when running the auto-join feature.

...

This change is sufficient because the main motivation behind not completing the RPC until the new voter set was committed was for an intuitive UX for the operator, since adding voters was done manually. The inactive observer controllers that send AddRaftVoterRequest  as a part of auto-join do not care if the new voter set was committed, since they will retry the request on a timer until it completes successfully.

...

--- a/clients/src/main/resources/common/message/AddRaftVoterRequest.json
+++ b/clients/src/main/resources/common/message/AddRaftVoterRequest.json
@@ -18,7 +18,8 @@
   "type": "request",
   "listeners": ["controller", "broker"],
   "name": "AddRaftVoterRequest",
-  "validVersions": "0",
+  // Version 1 adds the AckWhenCommitted field.
+  "validVersions": "0-1",
   "flexibleVersions": "0+",
   "fields": [
     { "name": "ClusterId", "type": "string", "versions": "0+", "nullableVersions": "0+",
@@ -37,6 +38,8 @@
         "about": "The hostname." },
       { "name": "Port", "type": "uint16", "versions": "0+",
         "about": "The port." }
-    ]}
+    ]},
+    { "name": "AckWhenCommitted", "type": "bool", "versions": "1+", "default": "true",
+      "about": "When true, return a response after the new voter set is committed. Otherwise, return after the leader writes the changes locally." }
    ]

The default value of AckWhenCommitted  will be true to preserve the intuitive UX of the RPC for the operator. When AckWhenCommitted  is set to false , the AddRaftVoterRequest  RPC does not guarantee that the new voter set has been committed upon receiving a response, and may require retries.

Compatibility, Deprecation, and Migration Plan

...