DUE TO SPAM, SIGN-UP IS DISABLED. Goto Selfserve wiki signup and request an account.
...
Operators who have never overridden the broker config (i.e., it is implicitly `500`) need no action.
Test Plan
### Benchmark workstream
The proposed ceiling of `10,000` is an initial value chosen as 20× the current `500` ceiling — large enough to relieve the pressure described in the PR thread without entering territory where coordinator failover replay time becomes operationally painful. **A dedicated benchmarking sub-task will be filed under KAFKA-20070** to validate this choice with empirical data; the ceiling may be adjusted up or down before the KIP vote based on the results.
The benchmark sub-task should run two scenarios on a representative share-coordinator cluster:
1. **Sustained-write workload**: high `WriteShareGroupState` QPS against a fixed group, varying the per-group override across `{500, 2000, 5000, 10000, 25000, 50000}`. Measure: (a) `__share_group_state` log growth rate, (b) coordinator failover replay time, (c) write amplification ratio (snapshot-bytes / update-bytes).
2. **Pruning-lag workload**: same as above but track time from "old key state superseded" to "log segment containing it eligible for deletion." This is the metric that operationalizes chia7712's "unbounded growth" concern.
The final ceiling is the largest value for which (b) replay time and (a) log growth rate stay within currently observed operational SLOs. If benchmarks show `10,000` is unsafe, the KIP will be revised downward (e.g. to `5,000`) before vote; if benchmarks show headroom, it may be revised upward.
### Unit tests
- `ShareCoordinatorConfigTest`: assert validator accepts `0`, `500`, `NEW_MAX`; rejects `-1`, `NEW_MAX + 1`.
- New `GroupConfigTest` cases: per-group value `0` and any positive value `<= broker_ceiling` accepted; values `> broker_ceiling` rejected with the bound named in the message.
- `ShareCoordinatorShardTest`: when a group has the override set to `N`, `generateShareStateRecord` writes a `ShareSnapshot` after exactly `N` updates and a `ShareUpdate` before. When unset, falls back to broker default. Pre-existing tests at `ShareCoordinatorShardTest.java:757,888` continue to pass unchanged.
### Integration test
Extend `ShareCoordinatorIntegrationTest` (or add a new one) to:
1. Create two share groups on the same coordinator partition.
2. Set group A's override to a small value (e.g., 5), leave group B at the broker default.
3. Drive writes to both. Assert from the log that A produces snapshots ~5× more frequently than B.
### System test
Not required for v1. A backfill/restart system test asserting failover replay time as a function of the override is a reasonable follow-up.
Rejected Alternatives
Unit Tests - Extend ShareCoordinatorConfigTest, ShareCoordinatorShardTest and GroupConfigTest
Integration Tests - Extend ShareCoordinatorIntegrationTest
Rejected Alternatives
### 1. Raise broker max only; no per-group override
Considered. Smallest KIP, matches AndrewJSchofield's lighter framing. **Rejected** because it does not address chia7712's central concern: a single misconfigured group can block `__share_group_state` log pruning across all groups on its partition. Without a per-group knob, operators cannot safely tune for one workload without imposing the tuning on everyone.
A misconfigured group can write millions of updates without ever triggering an update (count-based) snapshot.
### 2. Per-group override only; keep broker max at 500.
The Considered. **Rejected** because the broker ceiling becomes the effective hard cap for any per-group value (the per-group config is `between(0, broker_value)`).
Leaving 500 in place forces operators to raise the broker config anyway to take advantage of per-group tuning, so coupling the two changes in one KIP is simpler.
### 3. Disallow `0` as a value (`atLeast(1)` or chia7712's `between(200, 1000)`)
Considered (PR #21291: AJ "we could consider disallowing it," chia7712 proposed `between(200, 1000)`). **Rejected** for this KIP because (a) the PR thread did not reach consensus, (b) treating `0` as misconfiguration is a behavior change orthogonal to the bound-evaluation work, and (c) doing it here would expand the KIP's risk surface. If benchmark data shows `0` causes pathological write amplification we will file a follow-up KIP.
### 4. Extend KIP-1240 or KIP-932 with this evaluation
Explicitly **rejected** by AndrewJSchofield on PR #21291: *"I would not increase the scope of KIP-1240. We can easily put in a specific KIP for this once we have done the work."* This KIP is that specific KIP.
### 5. Defer the new ceiling entirely (leave the number TBD until benchmarks complete)
Considered. **Rejected** because reviewers reading the KIP need a concrete proposal to react to. Instead, the KIP proposes `10,000` as the working ceiling and commits to the benchmarking sub-task under KAFKA-20070 as a gate before vote. The number will be revised in either direction if data warrants.
### 6. Transmit the per-group value in `WriteShareGroupState` requests
Keep the floor at `0` (raise only the ceiling)
The PR thread surfaced consensus that `0` and other very small values waste disk by writing a full snapshot for nearly every state change, and no real-world workload benefits from themConsidered as an alternative propagation mechanism. **Rejected** because the existing `GroupConfigManager` → `ShareGroupConfigProvider` channel already does exactly this for other share-group configs (`share.record.lock.duration.ms` etc.) and reusing it costs zero new RPC schema work.