Current state: Draft
Discussion thread:
JIRA:
Kafka Connect sink connectors consume from Kafka topics using traditional consumer groups.
In this model, each partition is exclusively assigned to one task. This creates two problems for Kafka-to-Kafka (and Kafka-to-external) pipelines:
1. Scaling is coupled to partition count.
If a topic has 12 partitions, you can run at most 12 sink tasks. I
ncreasing parallelism beyond the partition count requires repartitioning the source topic -- an operationally expensive and disruptive change.
2. Slow tasks block partitions.
If one sink task is slow (e.g., network latency to a downstream system), the records on its assigned partitions back up.
Other idle tasks cannot help because partition ownership is exclusive. This creates head-of-line blocking at the partition level.
3. Rebalance storms cause processing gaps.
When tasks are added, removed, or crash, consumer group rebalances revoke and reassign partitions.
During a rebalance, no task processes records from revoked partitions. With cooperative sticky rebalancing this is mitigated but not eliminated.
Share Groups (KIP-932) introduce queue semantics for Kafka consumers. Unlike consumer groups, Share Groups do not assign partitions exclusively.
Instead, records from a partition are acquired by any available consumer in the group. After processing, the consumer acknowledges the record (ACCEPT, RELEASE, ARCHIEVED, or REJECT).
This provides:
- Elastic scaling independent of partition count.
50 tasks can process a 12-partition topic because records are distributed at the record level, not the partition level.
- No head-of-line blocking.
If one task is slow, acquired records time out and are re-delivered to another task.
- No rebalance disruption.
Share Groups have no partition assignment protocol. Adding or removing tasks does not trigger reassignment of partitions.