Status

Current state: Under Discussion

Discussion thread: here [Change the link from the KIP proposal email archive to your own email thread]

JIRA: KAFKA-19400 - Getting issue details... STATUS

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

This KIP provides a simple solution to an availability issue exposed by the auto-join feature proposed in KIP-853 and how AddRaftVoterRequest  RPC is currently handled. The problem is because the active controller does not send a response to complete the AddRaftVoterRequest until after the new voter set is committed, and that KRaft (and Kafka in general) only support one in-flight request to a node. Consider the following scenario:

Some controller A that is automatically joining by sending the AddRaftVoterRequest  RPC is the same controller who's Fetch is needed to commit the new voter set. A clear example of this is when bootstrapping with --standalone and having controllers auto-join, as the first controller to auto-join will increase the voter set size from 1 to 2. The active controller needs controller A to complete a Fetch  RPC to complete the AddRaftVoterRequest  RPC. However, this case also applies when going from a voter set of size X to size X + 1, where a minority of X nodes from the old voter set are unavailable.

The reason this scenario causes unavailability is as follows: the current in-flight request for controller A is the AddRaftVoterRequest  RPC, which cannot complete until after controller A first replicates the new voter set, and sends another fetch to the leader. This state will expire the leader's checkQuorumTimer since the majority of nodes (which includes controller A) will not be able to fetch in time. Thus, the current implementation of AddRaftVoterRequest  will cause an unnecessary leadership failover and election when running the auto-join feature.

It is important to note that this scenario does not apply when handling the RemoveRaftVoterRequest  RPC in the auto-join feature because we need a majority of the new voter set to commit the new VotersRecord , which does not include the voter being removed.

Proposed Changes

The proposed change is to update the AddRaftVoterRequest  RPC with a boolean flag that tells the active controller whether the request was sent as a part of auto-join (i.e. sent from another controller), or not (e.g. sent via the AdminClient).

When the request is coming from another controller, the active controller will send a response after it appends the new voter set to its local log, rather than after that voter set is committed. This allows the "joining" replica to actually fetch the new voter set.

This change is sufficient because the main motivation behind not completing the RPC until the new voter set was committed was for an intuitive UX for the operator. The inactive controllers that send AddRaftVoterRequest  as a part of auto-join do not care if the new voter set was committed, since they will retry the request on a timer until it completes successfully.

Public Interfaces

Introduce a new version 1 to the AddRaftVoterRequest  RPC:

{
  "apiKey": 76,
  "type": "request",
  "listeners": ["controller", "broker"],
  "name": "AddVoterRequest",
  "validVersions": "0",
  "flexibleVersions": "0+",
  "fields": [
    { "name": "ClusterId", "type": "string", "versions": "0+" },
    { "name": "TimeoutMs", "type": "int32", "versions": "0+" },
    { "name": "TopicName", "type": "string", "versions": "0+", "entityType": "topicName",
      "about": "The name of the topic" },
    { "name": "TopicId", "type": "uuid", "versions": "0+",
      "about": "The unique topic ID" },
    { "name": "Partition", "type": "int32", "versions": "0+",
      "about": "The partition index" },
    { "name": "VoterId", "type": "int32", "versions": "0+",
      "about": "The replica id of the voter getting added to the topic partition" },
    { "name": "VoterDirectoryId", "type": "uuid", "versions": "0+",
      "about": "The directory id of the voter getting added to the topic partition" },
    { "name": "Listeners", "type": "[]Listener", "versions": "0+",
      "about": "The endpoints that can be used to communicate with the voter", "fields": [
      { "name": "Name", "type": "string", "versions": "0+", "mapKey": true,
        "about": "The name of the endpoint" },
      { "name": "Host", "type": "string", "versions": "0+",
        "about": "The hostname" },
      { "name": "Port", "type": "uint16", "versions": "0+",
        "about": "The port" }
    ]}
  ]
}

Compatibility, Deprecation, and Migration Plan

To make this change backwards compatible, we can make this field ignorable. This ensures compatibility between a controller that is adding itself with the new AddRaftVoterRequest via auto-join, and an old active controller that does not understand this field (i.e. does not have the auto-join feature). In that case, the unavailability issue described above is possible.

Test Plan

Add a unit test to test for compatibility.

Rejected Alternatives

We explored trying to get KRaft to support multiple in-flight requests, but there were some significant issues with this appraoch:

• Kafka doesn't support multiple in-flight requests on the "server" side. What this means exactly is that the receiver of a request mutes the connection of the socket it reads from, making it unable to process another in-flight request on that connection until it sends a response back for the first.
  • Adding support for this has much larger implications outside of KRaft and would require another KIP.
• Because of the point above, this means KRaft would need to establish 2 connections. One for AddRaftVoterRequest and one for essentially everything else. We found this solution to be overkill for our auto-join, and felt that hardcoding two connections is bad design.