Page History

...

Current state: Under Discussion

Discussion thread: here [Change the link from the KIP proposal email archive to your own email thread]

JIRA: here KAFKA-20427 [Change the link from KAFKA-1 to your own ticket]

...

When running an Apache Kafka cluster within a cloud-native environment based on Kubernetes and deployed by an operator like Strimzi, it’s not uncommon to rename DNS domains. In this case, when the Kubernetes cluster's DNS configuration changes (e.g., from cluster.local to cluster-new.local, or changing the default DNS suffix), all pod DNS names are affected. For example, controller FQDNs change from my-cluster-kafka-0.my-cluster-kafka-brokers.kafka-prod.svc.cluster.local to my-cluster-kafka-0.my-cluster-kafka-brokers.kafka-prod.svc.cluster-new.local. 

Furthermore, there is no safe recovery from majority loss. For example,   if  2 2 of 3 controllers are permanently gone, you cannot update the VotersRecord and must re-bootstrap with data loss, the metadata stored on those controllers (including leader epochs, ISR/ELR state) is lost. Kafka cannot recover from such metadata loss and any re-bootstrap attempt will require additional data loss beyond the initial metadata loss, as the system lacks the information needed to determine which data is authoritative and which replicas are safely in-sync. 

Finally, recovery from volume snapshot (i.e PVC in Kubernetes) is also fragile. It requires preserving the cluster name and the DNS domain at least, but if one of them changes, pods get new identities (common in Kubernetes), the VotersRecord in the metadata log no longer matches the reality, and there is no config file to simply edit to fix the mismatch. 

...