...

Current state: Draft

Discussion thread: here [Change the link from the KIP proposal email archive to your own email thread] 

JIRA: here [Change the link from KAFKA-1 to your own ticket]

...

Currently when handling topics or partitions creation requests, Kafka currently enforces all replicas to be created in order to fulfill the request. While all other functionalities (produce/consume) are fault tolerant and can handle some brokers down, topics and partitions creation stop working as soon as there are no enough replicas available. In small clusters, when one node is unavailable, for example when a broker is being restarted, it's possible that there are not enough online brokers to satisfy topic/partition creation even though there are enough brokers to satisfy the "min.insync.replicas" configuration of the topic.

...

We would like to allow users to create topics and partitions even when the current available number of brokers is less than the number of requested replicas. It would still require at least enough brokers to satisfy the  "min.insync.replicas" configuration of the topic. This guarantees producers will be able to use the topic/partition regardless of their acks setting, ie the lack of replicas will be invisible to all users. Because it's possible to create a topic with less replicas than "min.insync.replicas", the actual requirement will be min(min.insync.replicas, replicas).

When handling a CreateTopics or CreatePartitions request, in case not enough brokers are available to satisfy the replication factor, placeholder . Placeholder replica ids (-1, -2, -3) will be inserted for the missing replicas. In case a topic or partition is created under-replicated, the error code will still be NONE in the CreateTopics and CreatePartition responses.

When a new broker joins the cluster, the controller will check if any partitions have placeholders replicas ids and if so assign this broker as a replica (only if this broker is not already a replica).
This may lead to some topics/partitions not optimally spread across all racks, but note that this may already happen (without this KIP) when topics/partitions are created while all brokers in a rack are offline (ie: an availability zone is offline). Tracking topics/partitions not optimally spread across all racks can be tackled in a follow up KIP.

Because this feature can make partitions appear under replicated, it is disabled by default and can be enabled by a broker configuration: enable.under.replicated.topic.creation.
In case a topic or partition is created under-replicated, the error code will still be NONE in the CreateTopics and CreatePartition responses.

Public Interfaces

Broker Configuration

...