...

This page is meant as a template for writing a KIP. To create a KIP choose Tools->Copy on this page and modify with your content and replace the heading with the next KIP number and a description of your issue. Replace anything in italics with your own description.

Status

Current state: "Draft"

Discussion thread: here 

...

Configurable snapshot frequency for share groups

Motivation

Kafka has share groups. As multiple consumers in the same share group can consume from the same partition concurrently, records get individually acknowledged and broker tracks per record state(delivered, acked, in-flight etc).

...

Note : There is another config share.coordinator.cold.partition.snapshot.interval.ms (default 5 mins) which forces snapshotting on a timely basis, but only for share partitions with no updates. So the old log recs of the idle groups would be eligible for cleanup.

Public Interfaces

1. Updated broker-level config: share.coordinator.snapshot.update.records.per.snapshot

...

public static final String SHARE_SNAPSHOT_UPDATE_RECORDS_PER_SNAPSHOT_CONFIG = "share.snapshot.update.records.per.snapshot";

Proposed Changes

1. Change broker level lower and upper bounds

...

- Update the [Kafka Configuration](https://kafka.apache.org/documentation/#configuration) page to reflect the new broker ceiling and to add the new group-level entry under "Share Group Configurations."
- Update the share-group operator documentation (`docs/streams/...` equivalent for share groups, exact location to be confirmed during implementation) with a tuning section explaining the snapshot/update tradeoff.

Compatibility, Deprecation, and Migration Plan

Behavior break : floor raised from `0` to `200` -- any cluster currently configured with share.coordinator.snapshot.update.records.per.snapshot set to a value below 200 will fail broker startup config validation after upgrade. Clusters using the default 500 or any value in [200, 500] are unaffected. Clusters that previously set the value to anything between 501 and 1000 — not currently possible since the existing range is [0, 500] — would also pass validation in the new range; this case does not apply on upgrade.

...

Operators who have never overridden the broker config (i.e., it is implicitly `500`) need no action.

Test Plan

Unit Tests - Extend ShareCoordinatorConfigTest, ShareCoordinatorShardTest and GroupConfigTest

Integration Tests - Extend ShareCoordinatorIntegrationTest

Rejected Alternatives

1. Raise broker max only; no per-group override

...