...

This page is meant as a template for writing a KIP. To create a KIP choose Tools->Copy on this page and modify with your content and replace the heading with the next KIP number and a description of your issue. Replace anything in italics with your own description.

Status

Current state: [One of "Under Discussion", "Accepted", "Rejected"] Draft

Discussion thread: here [Change the link from the KIP proposal email archive to your own email thread]here 

JIRA: here [Change the link from KAFKA-1 to your own ticket]here 

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

...

without requiring third-party transaction managers, sink-side dedup conventions, or application-level compensation logic.

Public Interfaces

Briefly list any new interfaces that will be introduced as part of this proposal or any existing interfaces that will be removed or changed. The purpose of this section is to concisely call out the public contract that will come along with this feature.

A public interface is any change to the following:

• Binary log format

• The network protocol and api behavior

• Any class in the public packages under clientsConfiguration, especially client configuration

  • org/apache/kafka/common/serialization

  • org/apache/kafka/common

  • org/apache/kafka/common/errors

  • org/apache/kafka/clients/producer

  • org/apache/kafka/clients/consumer (eventually, once stable)

• Monitoring

• Command line tools and arguments

• Anything else that will likely break existing users in some way when they upgrade

Proposed Changes

Why the gap matters now

The most common multi-cluster deployment patterns in production Kafka today all need cross-cluster atomicity but cannot get it:

 1.  Cross-region active-active mirroring. Records produced to region A must appear in region B exactly once, with consumer-offset state on each region kept consistent with the data. MirrorMaker 2 provides at-least-once delivery with offset translation, but consumers must implement their own deduplication if they care about EOS. There is no Kafka-native solution.
 2.  Cross-cluster CTP (consume-transform-produce). An application reads from a source cluster, transforms records, and produces to a destination cluster. Today this requires either (a) abandoning Kafka transactions entirely and implementing application-level idempotency keys, or (b) misusing producer.sendOffsetsToTransaction with a ConsumerGroupMetadata from a foreign cluster — which silently produces incorrect results because the offsets are written to the wrong cluster's __consumer_offsets.
 3.  Hybrid sink topologies. A workload that fans records from one source cluster to multiple destination clusters (e.g. a primary plus a DR replica, or a sharded write path) has no atomic mechanism today to ensure either all destinations or none receive a given record.
 4.  Heterogeneous transaction boundaries. Applications that combine a Kafka transaction with a non-Kafka external system (e.g. a database) have KIP-939 as a 2PC primitive — but applications that combine two Kafka transactions in different clusters have nothing. The latter is structurally simpler than the former, yet remains unsupported. [Future Scope]

In every case, application developers today resort to the same workarounds: idempotent producers with embedded dedup keys, application-level inbox tables, custom mirror workers.

Public Interfaces

Thinking...

Proposed Changes

Thinking..Describe the new thing you want to do in appropriate detail. This may be fairly extensive and have large subsections of its own. Or it may be a few sentences. Use judgement based on the scope of the change.

Compatibility, Deprecation, and Migration Plan

• What impact (if any) will there be on existing users?
• If we are changing behavior how will we phase out the older behavior?
• If we need special migration tools, describe them here.
• When will we remove the existing behavior?

Test Plan

Describe in few sentences how the KIP will be tested. We are mostly interested in system tests (since unit-tests are specific to implementation details). How will we know that the implementation works as expected? How will we know nothing broke?

Rejected Alternatives

Thinking...

Test Plan

Thinking...

Rejected Alternatives

Thinking..If there are alternative ways of accomplishing the same thing, what were they? The purpose of this section is to motivate why the design is the way it is and not some other way.