...

• Consume‑Transform‑Produce (CTP): bind acks to Kafka output transaction.
• Consume‑only frameworks: transactional acks independent of producer.

Frameworks:

- Apache Flink: Exactly-once checkpointing with Share Group sources
- Apache Spark: Structured Streaming with Share Group consumers
- Any coordinator-worker streaming framework requiring atomic acknowledgements

Although in this KIP we are refering Flink Stream processing engine few places but it is valid for any stream processing engine - almost all have same pattern of offset commits or replay when something crashes. 

1.1 Background: Share Groups

...

What changes is which partitions are added to the transaction:

The WriteTxnMarkers request dispatches commit/abort markers to all partitions in the transaction set. If only sendShareAcksToTransaction() was called, markers go only to __share_group_state. If both sendOffsetsToTransaction() and sendShareAcksToTransaction() were called in the same transaction, markers go to both. The transaction coordinator does not distinguish between these — it just tracks partition sets.

...

CLIENT --> BROKER communication (over the Kafka wire protocol):

  Share Consumer -----> Share Group Coordinator
    ShareFetch, ShareAcknowledge, ShareGroupHeartbeat

  TransactionManager --> Transaction Coordinator
    InitProducerId, AddPartitionsToTxn, AddShareAcksToTxn(NEW), EndTxn

  TransactionManager --> Share Group Coordinator
    TxnShareAcknowledgeRequest (NEW)

  Producer ------------> Data Partition Leaders
    ProduceRequest


BROKER --> BROKER communication (internal, not client-visible):

  Transaction Coordinator --> Data Partition Leaders
    WriteTxnMarkers (COMMIT/ABORT control record)

  Transaction Coordinator --> Group Coordinator (via __consumer_offsets leader)
    WriteTxnMarkers (materializes transactional offset commits)

  Transaction Coordinator --> Share Group Coordinator (via __share_group_state leader) [NEW]
    WriteTxnMarkers (materializes transactional share acks)

Abort case: 

6. Corner Cases

Case 1: Crash BEFORE beginTransaction()

poll() -> records [0..9] ACQUIRED
<CRASH>

What happens:

• No transaction was started, no output produced, no acks sent
• ACQUIRED locks on records 0-9 expire (controlled by share.record.lock.duration.ms)
• Broker re-delivers records 0-9 to another member of the share group
• No data loss, no duplicates

Case 2: Crash AFTER send() but BEFORE commitTransaction()

poll() -> records [0..9] ACQUIRED
beginTransaction()
send(output for 0-9)       ← records written to broker, but transactional
<CRASH>

What happens:

• Output records are in the partition log but invisible (read_committed consumers skip uncommitted)
• No EndTxnRequest was sent
• TC detects transaction timeout (transaction.timeout.ms), auto-aborts
• TC sends WriteTxnMarkers(ABORT) -> output records get ABORT marker, permanently invisible
• Share group: no acks were sent, ACQUIRED locks expire, records 0-9 re-delivered
• No data loss, no duplicates

Case 3: Crash AFTER sendShareAcksToTransaction() but BEFORE commitTransaction()

poll() -> records [0..9] ACQUIRED
beginTransaction()
send(output)
sendShareAcksToTransaction()  ← acks written to SGC as PENDING
<CRASH>

What happens:

• Output records: in log, invisible (uncommitted)
• Share acks: stored in SGC as PENDING (not materialized)
• TC detects timeout, sends WriteTxnMarkers(ABORT) to both data partitions AND __share_group_state
• Output: ABORT marker written, records permanently invisible
• Share acks: SGC discards pending acks, locks expire, records 0-9 re-delivered
• No data loss, no duplicates -- this is the critical case that KIP-1289 solves

Case 4: Crash AFTER EndTxn(COMMIT) sent but BEFORE WriteTxnMarkers completes

poll() -> records [0..9] ACQUIRED
beginTransaction()
send(output)
sendShareAcksToTransaction()
commitTransaction()            ← EndTxn sent, PREPARE_COMMIT logged
<TC crashes or broker restart>

What happens:

• PREPARE_COMMIT is durable in __transaction_state log
• When TC recovers (or new leader elected for __transaction_state partition), it replays the log
• TC sees PREPARE_COMMIT, resumes sending WriteTxnMarkers(COMMIT) to all partitions
• Output records become visible, share acks materialized
• No data loss, no duplicates -- the two-phase commit guarantees completion

Case 5: ProducerFencedException (zombie detection)

Instance A: beginTransaction(), send(), ...
Instance B: initTransactions() with same transactional.id
            ← TC bumps epoch, A is now a zombie
Instance A: commitTransaction()
            ← TC rejects: ProducerFencedException

What happens:

• Instance A's transaction is aborted by the TC (epoch fenced)
• Any pending output and share acks from A are discarded
• Instance B takes over, re-processes the records
• No data loss, no duplicates -- fencing prevents split-brain

Case 6: abortTransaction() called explicitly

poll() -> records [0..9] ACQUIRED
beginTransaction()
send(output for 0-4)
record 5 fails validation
abortTransaction()             ← explicit abort

What happens:

• TC writes PREPARE_ABORT, sends WriteTxnMarkers(ABORT)
• Output records 0-4: ABORT marker, permanently invisible
• Share acks (if any sent): discarded
• ACQUIRED locks expire, all records 0-9 re-delivered
• Application gets fresh batch, can retry
• No data loss

Case 7: Partial produce failure (network error to one partition)

beginTransaction()
send("enriched-orders-0", rec1)   ← success
send("enriched-orders-1", rec2)   ← network error, KafkaException

What happens:

• Application catches KafkaException, calls abortTransaction()
• Same as Case 6: everything rolled back, records re-delivered
• No data loss

The Guarantee Matrix

The fundamental property: at no point can output records be visible while share acks are uncommitted, or vice versa. They are in the same transaction and share the same COMMIT/ABORT fate.

7. Compatibility, Deprecation, and Migration Plan

...