DUE TO SPAM, SIGN-UP IS DISABLED. Goto Selfserve wiki signup and request an account.
...
JIRA:
| Jira | ||||||
|---|---|---|---|---|---|---|
|
1 Motivation
1.1 The Problem: Scaling Kafka-to-Kafka Pipelines Today
Kafka Connect sink connectors consume from Kafka topics using traditional consumer groups.
...
During a rebalance, no task processes records from revoked partitions. With cooperative sticky rebalancing this is mitigated but not eliminated.
1.2 How Share Groups Solve This
Share Groups (KIP-932) introduce queue semantics for Kafka consumers. Unlike consumer groups, Share Groups do not assign partitions exclusively.
...
Share Groups have no partition assignment protocol. Adding or removing tasks does not trigger reassignment of partitions.
1.3 Kafka Connect as the Natural Integration Point
Kafka Connect is the standard framework for building data pipelines into and out of Kafka. Integrating Share Groups into Connect's sink connector runtime gives every existing sink connector access to queue semantics **with a configuration change only** -- no connector code modifications required.
1.4 Delivery Semantics
| Mode | Guarantee | Mechanism |
|---|---|---|
| At-least-once(default) | Every record is delivered at least once; duplicates possible on failure | ShareConsumer.acknowledge(ACCEPT) after successful task.put(); records released on failure for re-delivery |
| Exactly-once | Every record is delivered exactly once | KIP-1289 transactional acknowledgments: producer.sendShareAcksToTransaction() binds acknowledgments to the output transaction |
At-least-once is the initial target. Exactly-once requires KIP-1289 (Transactional Acknowledgments for Share Groups) to be implemented and is described as a future phase.
1.5 Error Handling / Delivery Semantics
This KIP integrates with KIP-1191. In Share Group mode, Connect uses `AcknowledgeType.REJECT` for fatal errors and relies on the broker-side DLQ configured by KIP-1191. Retriable failures use `AcknowledgeType.RELEASE`. If a DLQ is configured for the share group, the broker is the single source of DLQ records; Connect does not emit its own DLQ records in this mode.
EO Constraints
For exactly-once, records with pending transactional acknowledgments must not be re-delivered while the transaction is open; KIP-1289 must suppress or renew acquisition locks until commit/abort. Operationally, `share.acquisition.lock.timeout.ms` must exceed worst-case `task.put()` plus transaction commit latency, otherwise duplicates are possible even with EOS.