Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

3.1 New Configuration Properties

PropertyTypeDefaultDescription
consumer.group.protocolstringconsumerExisting property. When set to share, the Connect worker creates a KafkaShareConsumer instead of a KafkaConsumer for sink tasks.

3.1.1 Worker-level configuration (`connect-distributed.properties`)


PropertyTypeDefaultDescription
consumer.override.group.protocolstring(inherited from worker)Per-connector override. Set to share to opt a single connector into queue semantics.
share.group.idstringconnect-<connector-name>The Share Group ID. Defaults to the same naming convention as consumer groups.
share.acknowledgement.modestringexplicitexplicit: worker calls acknowledge(ACCEPT) after task.put() succeeds. implicit: acknowledgments are sent on the next poll() (simpler, lower latency, weaker guarantee).
share.acquisition.lock.timeout.msint30000Maximum time a record remains in ACQUIRED state before the broker releases it for re-delivery. Must be greater than the expected task.put() latency.
share.delivery.semanticsstringat-least-onceat-least-once or exactly-once. Exactly-once requires KIP-1289 and a transactional producer.
share.max.delivery.countint5Maximum number of times a record can be re-delivered before being sent to the Dead Letter Queue (if configured). Maps to Share Group's group.share.record.lock.partition.limit.


3.1.2 Connector-level configuration (per-connector JSON)

...

The difference is entirely in the worker runtime:

AspectWorkerSinkTask (today)WorkerShareSinkTask (proposed)
ConsumerKafkaConsumerKafkaShareConsumer
Subscriptionconsumer.subscribe(topics, rebalanceListener)shareConsumer.subscribe(topics)
Pollconsumer.poll()shareConsumer.poll()
Offset trackingcurrentOffsets map + consumer.commitSync()Per-record shareConsumer.acknowledge(record, ACCEPT) + shareConsumer.commitSync()
RebalanceConsumerRebalanceListener calling task.open()/close()No rebalances. task.open() called once at startup for all subscribed topics.
Failure handlingRetriableException -> pause consumer, retry batchRetriableException -> acknowledge(RELEASE) for batch, records re-delivered by broker

4. Compatibility, Deprecation, and Migration Plan

...