Page History

...

Ensure atomicity between share‑group acknowledgments and downstream side effects (e.g., Kafka transactions) so a record is either both processed and acknowledged or neither.

1. Share consumer polls records -> broker sets them to ACQUIRED.
2. Framework processes and acknowledges records (implicit or explicit).
3. Checkpoint fails before sink outputs are committed.
4. Records are permanently ACKNOWLEDGED (not redelivered) -> data loss.
5. Note:
   1. Share records become terminal when acked: RecordState
   2. Ack path today is irreversible once ACKNOWLEDGED: SharePartition.acknowledge()

Goal: Enable exactly-once read semantics via transactional acknowledgements.

We will be following the similar existing pattern we have in Kafka Producer Transactions.

Use Cases

• Consume‑Transform‑Produce (CTP): bind acks to Kafka output transaction.
• Consume‑only frameworks: transactional acks independent of producer.

Frameworks:

- Apache Flink: Exactly-once checkpointing with Share Group sources
- Apache Spark: Structured Streaming with Share Group consumers
- Any coordinator-worker streaming framework requiring atomic acknowledgements

...