Versions Compared

Old Version 27

changes.mady.by.user Shekhar Rajak

Saved on

compared with

New Version 28

changes.mady.by.user Shekhar Rajak

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

- Apache Flink: Exactly-once checkpointing with Share Group sources
- Apache Spark: Structured Streaming with Share Group consumers
- Any coordinator-worker streaming framework requiring atomic acknowledgements

Public Interfaces


New APIMirrorsWhy Needed
sendShareAcksToTransaction()sendOffsetsToTransaction()Acks are stored in __share_group_state, not __consumer_offsets
AddShareAcksToTxnRequestAddOffsetsToTxnRequestTransaction coordinator must track __share_group_state partitions
TxnShareAcknowledgeRequestTxnOffsetCommitRequestAck semantics are per‑record state, not per‑offset

APIs 

  • KafkaProducer.sendShareAcksToTransaction(acks, groupId) - for CTP 
    • This mirrors the existing KafkaProducer.sendOffsetsToTransaction(offsets, groupMetadata)

...

Compatibility, Deprecation, and Migration Plan

Impact on Existing Users

  • If older broker doesn’t support TxnShareAcknowledgeRequest, fallback to current explicit/implicit ack with warning.
  • Clients must fail fast on unsupported brokers
  • No breaking changes for existing Share Group users
  • `transactional` mode is opt-in via `share.acknowledgement.mode` config
  • Existing `implicit` and `explicit` modes continue to work unchanged

Migration Path

1. Phase 1: Add `transactional` acknowledgement mode (backward compatible)
2. Phase 2: Streaming frameworks (Flink, Spark) implement transactional sources
3. Phase 3: Documentation and best practices for exactly-once semantics

...

- Broker failure during PREPARE state
- Coordinator crash before/after prepare
- Consumer crash during transaction
- Network partitions between coordinator and brokers

Include exactly‑once recovery tests similar to Flink e2e:

    • Kill after sendShareAcksToTransaction but before commit.
    • Kill after commit but before notifyCheckpointComplete.
    • Ensure acks are either replayed or visible, never lost.

Rejected Alternatives

If there are alternative ways of accomplishing the same thing, what were they? The purpose of this section is to motivate why the design is the way it is and not some other way.