Page History

...

Current state: Under Discussion

Discussion thread: here [Change the link from the KIP proposal email archive to your own email thread]

JIRA: here KAFKA-20427 [Change the link from KAFKA-1 to your own ticket]

...

When running an Apache Kafka cluster within a cloud-native environment based on Kubernetes and deployed by an operator like Strimzi, it’s not uncommon to rename DNS domains. In this case, when the Kubernetes cluster's DNS configuration changes (e.g., from cluster.local to cluster-new.local, or changing the default DNS suffix), all pod DNS names are affected. For example, controller FQDNs change from my-cluster-kafka-0.my-cluster-kafka-brokers.kafka-prod.svc.cluster.local to my-cluster-kafka-0.my-cluster-kafka-brokers.kafka-prod.svc.cluster-new.local. 

Furthermore, there is no safe recovery from majority loss. For example,   if  2 2 of 3 controllers are permanently gone, you cannot update the VotersRecord and must re-bootstrap with data lossthe metadata stored on those controllers (including leader epochs, ISR/ELR state) is lost. Kafka cannot recover from such metadata loss and any re-bootstrap attempt will require additional data loss beyond the initial metadata loss, as the system lacks the information needed to determine which data is authoritative and which replicas are safely in-sync. 

Finally, recovery from volume snapshot (i.e PVC in Kubernetes) is also fragile. It requires preserving the cluster name and the DNS domain at least, but if one of them changes, pods get new identities (common in Kubernetes), the VotersRecord in the metadata log no longer matches the reality, and there is no config file to simply edit to fix the mismatch. 

...

Finally, in case of rejected changes because the user is trying to change the voter IDs, the output will be: 

Storage directory /var/kafka/kraft-logs is already formatted. 

Override voters mode enabled, checking if VoterSet needs updating... 

... 

Changes detected: 

VoterSetDiff{hasVoterIdChanges=true, hasDirectoryIdChanges=false, endpointChanges={}} 

Error: --override-voters cannot be used for changing node IDs or directory IDs. 

Changes detected: 

VoterSetDiff{hasVoterIdChanges=true, hasDirectoryIdChanges=false, endpointChanges={}} 

When the storage is not formatted yet, the --override-voters voters doesn’t have any additional impact, but the formatting runs as usual. It also doesn’t have any effect if the persisted voter set is the same as the one provided via the --initial-controllers option.

...