DUE TO SPAM, SIGN-UP IS DISABLED. Goto Selfserve wiki signup and request an account.
...
Note: The existing `SinkTask` interface is **is not modified**. Connectors do not need code changes. The `put(Collection<SinkRecord>)` contract remains the same.
...
This keeps them co-located with the existing `sink-record-read-total`, `sink-record-send-total`, etc. and avoids a separate metric namespace.
| Sensor Name | Metric Name | Type | Traditional Consumer (group.protocol=consumer) | Share Consumer (group.protocol=share) |
sink-record-acquire | sink-record-acquire-rate | Rate | not registered | Records/sec acquired from the share group |
sink-record-acquire-total | CumulativeSum | not registered | Total records acquired from the share group | |
sink-record-acknowledge | sink-record-acknowledge-rate | Rate | not registered | Records/sec acknowledged (ACCEPT) |
sink-record-acknowledge-total | CumulativeSum | not registered | Total records acknowledged (ACCEPT) | |
sink-record-release | sink-record-release-rate | Rate | not registered | Records/sec released (RELEASE) for re-delivery |
sink-record-release-total | CumulativeSum | not registered | Total records released for re-delivery | |
sink-record-reject | sink-record-reject-rate | Rate | not registered | Records/sec rejected (REJECT) to DLQ |
sink-record-reject-total | CumulativeSum | not registered | Total records rejected to DLQ | |
acknowledge-time | acknowledge-time-max | Max | not registered | Max time (ms) between poll() and acknowledge() |
acknowledge-time-avg | Avg | not registered | Avg time (ms) between poll() and acknowledge() | |
sink-record-redelivery | sink-record-redelivery-total | CumulativeSum | not registered | Total records with delivery count > 1 |
Conversely, the following existing `WorkerSinkTask` sensors have no Share Group equivalent and are not registered by `WorkerShareSinkTask`:
| Existing Sensor | Why not applicable to Share Groups |
partition-count | Share Groups don't assign partitions exclusively to tasks. All tasks consume from all subscribed partitions. |
offset-seq-number | Share Groups don't use consumer offsets. Acknowledgments replace offset commits. |
offset-commit-completion | No offset commits in Share Groups. Replaced by sink-record-acknowledge. |
offset-commit-completion-skip | No offset commits to skip. |
The existing sensors that are shared between both task types:
| Sensor | Behavior |
sink-record-read | Registered by both. Counts records polled (same semantics). |
sink-record-send | Registered by both. Counts records delivered to task.put(). |
sink-record-active-count | Registered by both. In Share Groups, this is the number of records currently ACQUIRED but not yet acknowledged. |
put-batch-time | Registered by both. Time spent in task.put(). |
4. Compatibility, Deprecation, and Migration Plan
4.1 Impact on Existing Users
- No impact by default. The default `group.protocol` remains `consumer` (standard consumer group). Existing connectors continue to work identically.
- Opt-in only. Share Groups are enabled per-connector or per-worker via configuration.
- No connector code changes required. The `SinkTask` interface is unchanged. Any existing sink connector works with Share Groups without modification.
4.2 Migration Path
1. Pre-requisite: Kafka broker version must support Share Groups (4.0+).
2. Enable at worker level: Set `consumer.group.protocol=share` in `connect-distributed.properties` to make all sink connectors use Share Groups.
3. Or enable per-connector: Set `consumer.override.group.protocol=share` in the connector config JSON.
4. Tune acquisition lock timeout: Set `share.acquisition.lock.timeout.ms` to a value greater than the expected `task.put()` latency. The default of 30 seconds is suitable for most workloads.
5. Monitor: Use the new `share-sink-task.*` metrics to observe acknowledgment patterns and re-delivery rates.
4.3 Rollback
To revert, remove the `group.protocol=share` configuration. The connector will resume using standard consumer groups.
Note that Share Groups and consumer groups maintain separate offset tracking, so the consumer group will resume from its last committed offset
(which may be behind the Share Group's position).
4.4 Deprecation
No existing features are deprecated. This is purely additive.
5. Test Plan
6. Future Work
...