...

without requiring third-party transaction managers, sink-side dedup conventions, or application-level compensation logic.

Public Interfaces

Briefly list any new interfaces that will be introduced as part of this proposal or any existing interfaces that will be removed or changed. The purpose of this section is to concisely call out the public contract that will come along with this feature.

A public interface is any change to the following:

• Binary log format

• The network protocol and api behavior

• Any class in the public packages under clientsConfiguration, especially client configuration

  • org/apache/kafka/common/serialization

  • org/apache/kafka/common

  • org/apache/kafka/common/errors

  • org/apache/kafka/clients/producer

  • org/apache/kafka/clients/consumer (eventually, once stable)

• Monitoring

• Command line tools and arguments

• Anything else that will likely break existing users in some way when they upgrade

Proposed Changes

Why the gap matters now

The most common multi-cluster deployment patterns in production Kafka today all need cross-cluster atomicity but cannot get it:

 1.  Cross-region active-active mirroring. Records produced to region A must appear in region B exactly once, with consumer-offset state on each region kept consistent with the data. MirrorMaker 2 provides at-least-once delivery with offset translation, but consumers must implement their own deduplication if they care about EOS. There is no Kafka-native solution.
 2.  Cross-cluster CTP (consume-transform-produce). An application reads from a source cluster, transforms records, and produces to a destination cluster. Today this requires either (a) abandoning Kafka transactions entirely and implementing application-level idempotency keys, or (b) misusing producer.sendOffsetsToTransaction with a ConsumerGroupMetadata from a foreign cluster — which silently produces incorrect results because the offsets are written to the wrong cluster's __consumer_offsets.
 3.  Hybrid sink topologies. A workload that fans records from one source cluster to multiple destination clusters (e.g. a primary plus a DR replica, or a sharded write path) has no atomic mechanism today to ensure either all destinations or none receive a given record.
 4.  Heterogeneous transaction boundaries. Applications that combine a Kafka transaction with a non-Kafka external system (e.g. a database) have KIP-939 as a 2PC primitive — but applications that combine two Kafka transactions in different clusters have nothing. The latter is structurally simpler than the former, yet remains unsupported. [Future Scope]

In every case, application developers today resort to the same workarounds: idempotent producers with embedded dedup keys, application-level inbox tables, custom mirror workers.

Public Interfaces

Thinking...

Proposed Changes

Thinking...Describe the new thing you want to do in appropriate detail. This may be fairly extensive and have large subsections of its own. Or it may be a few sentences. Use judgement based on the scope of the change.

Compatibility, Deprecation, and Migration Plan

...

Thinking...

Test Plan

Describe in few sentences how the KIP will be tested. We are mostly interested in system tests (since unit-tests are specific to implementation details). How will we know that the implementation works as expected? How will we know nothing broke?

Rejected Alternatives

Thinking...

Rejected Alternatives

Thinking..If there are alternative ways of accomplishing the same thing, what were they? The purpose of this section is to motivate why the design is the way it is and not some other way.