This page is meant as a template for writing a KIP. To create a KIP choose Tools->Copy on this page and modify with your content and replace the heading with the next KIP number and a description of your issue. Replace anything in italics with your own description.
Current state: Draft
Discussion thread: here
JIRA: here
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
Apache Kafka provides exactly-once semantics (EOS) only within the boundary of a single cluster. The transactional protocol introduced by KIP-98, hardened by KIP-890,
extended for external 2PC participation by KIP-939, (and abstracted into a first-class client object by KIP-1310 - in progress), is in every case scoped to one transaction coordinator on one cluster.
There is no native mechanism in Kafka today for an application to atomically commit a write to cluster A together with a write to cluster B, or to atomically advance a consumer offset on cluster A together with a produce to cluster B.
This KIP proposes a Global Transaction Coordinator (GTC) — a new optional Kafka role, deployable on existing brokers via process.roles=...,gtc —
that orchestrates a two-phase commit protocol across two or more participating Kafka clusters.
The proposal composes existing primitives (prepareTransaction / completeTransaction, rather than reinventing them.
Participating clusters opt in via a single configuration flag and remain individually unaware of one another; coordination flows exclusively through the GTC.
The result is a Kafka-native, end-to-end EOS guarantee for read-process-write workloads, mirroring topologies, and multi-cluster fan-out —
without requiring third-party transaction managers, sink-side dedup conventions, or application-level compensation logic.
The most common multi-cluster deployment patterns in production Kafka today all need cross-cluster atomicity but cannot get it:
1. Cross-region active-active mirroring. Records produced to region A must appear in region B exactly once, with consumer-offset state on each region kept consistent with the data. MirrorMaker 2 provides at-least-once delivery with offset translation, but consumers must implement their own deduplication if they care about EOS. There is no Kafka-native solution.
2. Cross-cluster CTP (consume-transform-produce). An application reads from a source cluster, transforms records, and produces to a destination cluster. Today this requires either (a) abandoning Kafka transactions entirely and implementing application-level idempotency keys, or (b) misusing producer.sendOffsetsToTransaction with a ConsumerGroupMetadata from a foreign cluster — which silently produces incorrect results because the offsets are written to the wrong cluster's __consumer_offsets.
3. Hybrid sink topologies. A workload that fans records from one source cluster to multiple destination clusters (e.g. a primary plus a DR replica, or a sharded write path) has no atomic mechanism today to ensure either all destinations or none receive a given record.
4. Heterogeneous transaction boundaries. Applications that combine a Kafka transaction with a non-Kafka external system (e.g. a database) have KIP-939 as a 2PC primitive — but applications that combine two Kafka transactions in different clusters have nothing. The latter is structurally simpler than the former, yet remains unsupported. [Future Scope]
In every case, application developers today resort to the same workarounds: idempotent producers with embedded dedup keys, application-level inbox tables, custom mirror workers.
Thinking...
Thinking...
Thinking...
Thinking...
Thinking...