Status

Current state: "DraftUnder Discussion"

Discussion thread: here 

JIRA: here 

Motivation

Kafka has share groups. As multiple consumers in the same share group can consume from the same partition concurrently, records get individually acknowledged and broker tracks per record state(delivered, acked, in-flight etc).

...

Compatibility, Deprecation, and Migration Plan

Behavior break

 floor  Floor raised from `0` to `200` '0' to '200' -- any cluster currently configured with share.coordinator.snapshot.update.records.per.snapshot set to a value below 200 will fail broker startup config validation after upgrade. Clusters using the default 500 or any value in [200, 500] are unaffected. Clusters that previously set the value to anything between 501 and 1000 — not currently possible since the existing range is [0, 500] — would also pass validation in the new range; this case does not apply on upgrade.

...

• Values 0–199: fail to start after upgrade (operator must fix)
• Values 200–500: unaffected
•  Values 501+: don't exist (can't happen)

Behavioral compatibility

(ceiling raise): Ceiling raise -- raising the ceiling from 500 to 1000 is non-breaking on its own. Brokers that retain their existing setting in [200, 500] behave identically to today. Operators can opt into the higher range explicitly.

...

Operators who have never overridden the broker config (i.e., it is implicitly `500`) need no action.

Test Plan

Unit Tests

ShareCoordinatorConfigTest (covers the broker-level bounds change)

ShareCoordinatorShardTest (covers the per-group config)

GroupConfigTest (covers the runtime use of the per-group value)

Integration Tests

 ShareCoordinatorIntegrationTest (covers end-to-end propagation of the per-group config) - Extend ShareCoordinatorConfigTest, ShareCoordinatorShardTest and GroupConfigTestIntegration Tests - Extend ShareCoordinatorIntegrationTest

Rejected Alternatives

Here are the rejected alternatives.

Raise broker max only

...

. No per-group override

A single broker-level setting forces all share groups on the cluster to use the same value. Different groups have different write profiles. A low-traffic group benefits from a low value (fast recovery), while a high-traffic group benefits from a high value. Without a per-group override, operators must pick one value that compromises for all groups.
Additionally, since all groups on the same __share_group_state partition share log-pruning behavior, one group set very high can delay pruning for unrelated groups on the same partition.

Per-group override only

...

. Keep broker max at 500. 

The broker ceiling acts as a hard cap on every per-group value (per-group is between(200, broker_value)). If we keep the broker max at 500 means, per-group overrides cannot exceed 500 either, so high-traffic groups cannot benefit from the per-group config. In this case operators would have to raise the broker ceiling immediately anyway to use the new per-group setting meaningfully. Basically the per-group config is not of much use without the ceiling raise.

Keep the floor at

...

'0; (raise only the ceiling)

Raising only the ceiling (between(0, 1000)) is of minimum risk and it is intact with the documented 0 semantics avoids any upgrade-time validation failure. But based on the PR thread discussion , there is a consensus that 0 and other very small values waste disk by writing a full snapshot for nearly every state change. 

...