DUE TO SPAM, SIGN-UP IS DISABLED. Goto Selfserve wiki signup and request an account.
| Table of Contents |
|---|
This page is meant as a template for writing a KIP. To create a KIP choose Tools->Copy on this page and modify with your content and replace the heading with the next KIP number and a description of your issue. Replace anything in italics with your own description.
Status
Current state: Draft
...
Binary log format
The network protocol and api behavior
Any class in the public packages under clientsConfiguration, especially client configuration
org/apache/kafka/common/serialization
org/apache/kafka/common
org/apache/kafka/common/errors
org/apache/kafka/clients/producer
org/apache/kafka/clients/consumer (eventually, once stable)
Monitoring
Command line tools and arguments
- Anything else that will likely break existing users in some way when they upgrade
New Broker Configuration
Configuration | Description | Default | Type |
min.insync.racks | When a producer sets acks to "all" (or "-1"), this configuration specifies the minimum number of distinct racks that must have in-sync replicas to acknowledge a produce request. If this minimum cannot be met, the broker will respond with NotEnoughRacksException. This works in conjunction with min.insync.replicas — both thresholds must be satisfied for a produce request to succeed. Insufficient replicas will raise NotEnoughReplicasException; insufficient rack diversity will raise NotEnoughRacksException. This ensures that acknowledged messages are durably replicated across multiple fault domains (e.g. availability zones or data centers), reducing the risk of data loss during a full rack failure. Requires broker.rack to be configured on all brokers. A value of 1 (default) disables rack-aware acknowledgement checking. | 1 | int |
New Error Message
| Error Message | Description |
|---|---|
| NOT_ENOUGH_RACKS | The number of distinct racks with in-sync replicas for the partition is less than the required minimum (min.insync.racks). The produce request cannot be satisfied until replicas on additional racks rejoin the ISR |
New Monitoring
| Metric | Scope | Description |
|---|---|---|
| UnderMinRackIsr | Partition | 1 when this partition's ISR spans fewer racks than min.insync.racks |
| AtMinRackIsr | Partition | 1 when this partition's ISR rack count equals exactly min.insync.racks |
| UnderMinRackIsrPartitionCount | Broker | count of leader partitions where isUnderMinRackIsr is true |
| AtMinRackIsrPartitionCount | Broker | count of leader partitions where isAtMinRackIsr is true |
Proposed Changes
Overview
The proposed change modifies the produce acknowledgement path in the broker to add a rack diversity check alongside the existing min.insync.replicas check. The change is minimal and confined to the ISR validation logic in the ReplicaManager.
...
Compatibility, Deprecation, and Migration Plan
...
...
This change is fully backward compatible. The new min.insync.racks configuration defaults to 1, which disables rack-aware acknowledgement checking. Existing clusters will see no change in behaviour until an operator explicitly sets min.insync.racks > 1.
...
No deprecations are introduced. The existing min.insync.replicas configuration continues to function exactly as it does today. The two configurations are complementary and both must be satisfied for a produce request to succeed.
Test Plan
...
System Tests
tests will deploy a multi-broker cluster across 3 simulated racks and verify:
Produces with acks=all succeed when ISR spans the required number of racks.
Produces with acks=all are rejected with NotEnoughRacksException when an entire rack is shut down and the ISR no longer meets min.insync.racks.
Dynamically lowering min.insync.racks to 1 during a rack outage allows produces to resume without a broker restart.
Both min.insync.replicas and min.insync.racks must be satisfied independently; failing either threshold rejects the produce with the correct error type.
Clusters with min.insync.racks=1 (default) exhibit no change in behaviour compared to a cluster without this feature.
Restoring the downed rack allows ISR to recover, and produces succeed again after restoring the original min.insync.racks value.
Rejected Alternatives
Implement Observer Nodes
...