This page is meant as a template for writing a KIP. To create a KIP choose Tools->Copy on this page and modify with your content and replace the heading with the next KIP number and a description of your issue. Replace anything in italics with your own description.
Current state: Under Discussion
Discussion thread: here [Change the link from the KIP proposal email archive to your own email thread]
JIRA: here
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
In a Raft-based quorum system, cluster membership changes (adding or removing voters) are applied through the log replication process. A membership change becomes effective only after it has been committed — that is, replicated to a majority of the current configuration. Between the moment a membership change entry is appended and the moment it is committed, the cluster exists in a transitional state where the leader’s in-memory voter list (voterSet) may include uncommitted changes.
By maintaining a separate committedVoterSet, the system ensures:
Safety: Quorum checks and leader decisions are based on a membership configuration that is guaranteed to be replicated and agreed upon by a majority.
Stability: The committed voter set changes only at commit points, avoiding transient or rolled-back configurations.
Correctness under Reconfiguration: Both during joint consensus transitions and simple add/remove operations, the system preserves Raft’s safety guarantees.
{
"apiKey": 55,
"type": "request",
"listeners": ["broker", "controller"],
"name": "DescribeQuorumRequest",
// Version 1 adds additional fields in the response. The request is unchanged (KIP-836).
// Version 2 adds additional fields in the response. The request is unchanged (KIP-853).
+ // Version 3 adds additional fields in the response. The request is unchanged.
+ "validVersions": "0-3",
"flexibleVersions": "0+",
"latestVersionUnstable": false,
"fields": [
{ "name": "Topics", "type": "[]TopicData", "versions": "0+",
"about": "The topics to describe.", "fields": [
{ "name": "TopicName", "type": "string", "versions": "0+", "entityType": "topicName",
"about": "The topic name." },
{ "name": "Partitions", "type": "[]PartitionData", "versions": "0+",
"about": "The partitions to describe.", "fields": [
{ "name": "PartitionIndex", "type": "int32", "versions": "0+",
"about": "The partition index." }
]
}]
}
]
} |
{
"apiKey": 55,
"type": "response",
"name": "DescribeQuorumResponse",
// Version 1 adds LastFetchTimeStamp and LastCaughtUpTimestamp in ReplicaState (KIP-836).
// Version 2 adds ErrorMessage, Nodes, ErrorMessage in ParitionData, ReplicaDirectoryId in ReplicaState (KIP-853).
+ // Version 3 adds CommittedVoters in PartitionData
+ "validVersions": "0-3",
"flexibleVersions": "0+",
"fields": [
{ "name": "ErrorCode", "type": "int16", "versions": "0+",
"about": "The top level error code."},
{ "name": "ErrorMessage", "type": "string", "versions": "2+", "nullableVersions": "2+", "ignorable": true,
"about": "The error message, or null if there was no error." },
{ "name": "Topics", "type": "[]TopicData",
"versions": "0+", "fields": [
{ "name": "TopicName", "type": "string", "versions": "0+", "entityType": "topicName",
"about": "The topic name." },
{ "name": "Partitions", "type": "[]PartitionData",
"versions": "0+", "fields": [
{ "name": "PartitionIndex", "type": "int32", "versions": "0+",
"about": "The partition index." },
{ "name": "ErrorCode", "type": "int16", "versions": "0+"},
{ "name": "ErrorMessage", "type": "string", "versions": "2+", "nullableVersions": "2+", "ignorable": true,
"about": "The error message, or null if there was no error." },
{ "name": "LeaderId", "type": "int32", "versions": "0+", "entityType": "brokerId",
"about": "The ID of the current leader or -1 if the leader is unknown."},
{ "name": "LeaderEpoch", "type": "int32", "versions": "0+",
"about": "The latest known leader epoch"},
{ "name": "HighWatermark", "type": "int64", "versions": "0+"},
{ "name": "CurrentVoters", "type": "[]ReplicaState", "versions": "0+" },
+ { "name": "CommittedVoters", "type": "[]ReplicaState", "versions": "3+", "ignorable": true , "about": "The voters has been committed."},
{ "name": "Observers", "type": "[]ReplicaState", "versions": "0+" }
]}
]},
{ "name": "Nodes", "type": "[]Node", "versions": "2+", "fields": [
{ "name": "NodeId", "type": "int32", "versions": "2+",
"mapKey": true, "entityType": "brokerId", "about": "The ID of the associated node" },
{ "name": "Listeners", "type": "[]Listener",
"about": "The listeners of this controller", "versions": "2+", "fields": [
{ "name": "Name", "type": "string", "versions": "2+", "mapKey": true,
"about": "The name of the endpoint" },
{ "name": "Host", "type": "string", "versions": "2+",
"about": "The hostname" },
{ "name": "Port", "type": "uint16", "versions": "2+",
"about": "The port" }
]}
]}
],
"commonStructs": [
{ "name": "ReplicaState", "versions": "0+", "fields": [
{ "name": "ReplicaId", "type": "int32", "versions": "0+", "entityType": "brokerId" },
{ "name": "ReplicaDirectoryId", "type": "uuid", "versions": "2+" },
{ "name": "LogEndOffset", "type": "int64", "versions": "0+",
"about": "The last known log end offset of the follower or -1 if it is unknown"},
{ "name": "LastFetchTimestamp", "type": "int64", "versions": "1+", "ignorable": true, "default": -1,
"about": "The last known leader wall clock time time when a follower fetched from the leader. This is reported as -1 both for the current leader or if it is unknown for a voter"},
{ "name": "LastCaughtUpTimestamp", "type": "int64", "versions": "1+", "ignorable": true, "default": -1,
"about": "The leader wall clock append time of the offset for which the follower made the most recent fetch request. This is reported as the current time for the leader and -1 if unknown for a voter"}
]}
]
} |
We also add new field committedVoters to QuorumInfo.
public class QuorumInfo {
private final int leaderId;
private final long leaderEpoch;
private final long highWatermark;
private final List<ReplicaState> voters;
+ private final List<ReplicaState> committedVoters;
private final List<ReplicaState> observers;
private final Map<Integer, Node> nodes;
QuorumInfo(
int leaderId,
long leaderEpoch,
long highWatermark,
List<ReplicaState> voters,
List<ReplicaState> committedVoters,
List<ReplicaState> observers,
Map<Integer, Node> nodes
) {
this.leaderId = leaderId;
this.leaderEpoch = leaderEpoch;
this.highWatermark = highWatermark;
this.voters = voters;
+ this.committedVoters = committedVoters;
this.observers = observers;
this.nodes = nodes;
}} |
This proposal introduces a new committed voter set tracking mechanism within the LeaderState to accurately represent the last committed quorum membership.
A new field committedVoterStates of type Map<Integer, ReplicaState> is added to the LeaderState class. This field stores the voter states.
Within the maybeUpdateHighWatermark() method, once the leader advances advance the HighWatermark, the committedVoterStates map is updated to a copy of the current active currentVoterStates. This ensures the committed voter set reflects the durable state of membership at the last committed log entry.
The distinction between currentVoterStates (potentially changing during ongoing elections or reconfigurations) and committedVoterStates (stable, committed membership) enables clear separation of transient membership changes from those that are confirmed and durable.
An accessor method committedVoterStates() is provided on LeaderState to expose this value. This method is used when constructing DescribeQuorumResponse messages to include the committed voter set, improving observability and operational diagnostics.
committedVoterStates may not be available. In these situations, we should read the committed voters from the replicatedLog using the replicatedLog#startOffset. However, since there is no information for logEndOffset, lastFetchTimestamp, or lastCaughtUpTimestamp, these fields will return their default value of -1.This mechanism strengthens the Raft membership model by making committed membership explicit and accessible to tooling and clients, thereby improving cluster state transparency.
DescribeQuorumRequest and DescribeQuorumResponse to a newer version, with backward compatibility preserved for the old protocol version.Unit test and integration test will be added.
If there are alternative ways of accomplishing the same thing, what were they? The purpose of this section is to motivate why the design is the way it is and not some other way.