Status

Current state: Draft

Discussion thread: here [Change the link from the KIP proposal email archive to your own email thread]

JIRA:

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

This KIP provides a simple solution to an availability issue exposed by the auto-join feature proposed in KIP-853 and how AddRaftVoterRequest  RPC is currently handled. The problem is because the active controller does not send a response to complete the AddRaftVoterRequest until after the new voter set is committed, and that KRaft (and Kafka in general) only support one in-flight request to a node. Consider the following scenario:

Some controller A that is automatically joining by sending the AddRaftVoterRequest  RPC is the same controller who's Fetch is needed to commit the new voter set. A clear example of this is when bootstrapping with --standalone and having controllers auto-join, as the first controller to auto-join will increase the voter set size from 1 to 2. The active controller needs controller A to complete a Fetch  RPC to complete the AddRaftVoterRequest  RPC. However, this case also applies when going from a voter set of size X to size X + 1, where a minority of X nodes from the old voter set are unavailable.

The reason this scenario causes unavailability is as follows: the current in-flight request for controller A is the AddRaftVoterRequest  RPC, which cannot complete until after controller A first replicates the new voter set, and sends another fetch to the leader. This state will expire the leader's checkQuorumTimer since the majority of nodes (which includes controller A) will not be able to fetch in time. Thus, the current implementation of AddRaftVoterRequest  will cause an unnecessary leadership failover and election when running the auto-join feature.

It is important to note that this scenario does not apply when handling the RemoveRaftVoterRequest  RPC in the auto-join feature because we need a majority of the new voter set to commit the new VotersRecord , which does not include the voter being removed.

Proposed Changes

The proposed change is to update the AddRaftVoterRequest  RPC with a boolean flag that tells the active controller whether the request was sent as a part of auto-join (i.e. sent from another controller), or not (e.g. sent via the AdminClient).

When the request is coming from another controller, the active controller will send a response after it appends the new voter set to its local log, rather than after that voter set is committed. This allows the "joining" replica to actually fetch the new voter set.

This change is sufficient because the main motivation behind not completing the RPC until the new voter set was committed was for an intuitive UX for the operator. The inactive controllers that send AddRaftVoterRequest  as a part of auto-join do not care if the new voter set was committed, since they will retry the request on a timer until it completes successfully.

Public Interfaces

Briefly list any new interfaces that will be introduced as part of this proposal or any existing interfaces that will be removed or changed. The purpose of this section is to concisely call out the public contract that will come along with this feature.

A public interface is any change to the following:

• Binary log format

• The network protocol and api behavior

• Any class in the public packages under clientsConfiguration, especially client configuration

  • org/apache/kafka/common/serialization

  • org/apache/kafka/common

  • org/apache/kafka/common/errors

  • org/apache/kafka/clients/producer

  • org/apache/kafka/clients/consumer (eventually, once stable)

• Monitoring

• Command line tools and arguments

• Anything else that will likely break existing users in some way when they upgrade

Compatibility, Deprecation, and Migration Plan

• What impact (if any) will there be on existing users?
• If we are changing behavior how will we phase out the older behavior?
• If we need special migration tools, describe them here.
• When will we remove the existing behavior?

Test Plan

Describe in few sentences how the KIP will be tested. We are mostly interested in system tests (since unit-tests are specific to implementation details). How will we know that the implementation works as expected? How will we know nothing broke?

Rejected Alternatives

If there are alternative ways of accomplishing the same thing, what were they? The purpose of this section is to motivate why the design is the way it is and not some other way.