Page History

...

While group.share.partition.max.record.locks correctly prevents broker memory pressure by bounding in-flight records, hitting that lock ceiling has a critical liveness consequence: fetching operations yield no further records from the primary partition. If the DLQ topic suffers a prolonged metadata or storage outage, the coordinator holds record locks until the limit is exhausted — stalling the primary pipeline entirely.

This re-introduces head-of-line blocking via a secondary infrastructure failure — precisely the problem Share Groups were originally designed to eliminate. A DLQ (a secondary concern) taking down the primary stream is an architectural single point of failure (SPOF). The safer operational posture is to prioritize system liveness over strict DLQ durability during infrastructure degradation.

Concrete example: A share group is processing 50,000 records/sec across 20 partitions. The DLQ topic's leader undergoes an unplanned election lasting 45 seconds. Without share.group.dlq.write.timeout.ms, the coordinator accumulates ARCHIVING-state record locks until group.share.partition.max.record.locks is exhausted. At that point, primary partition fetching halts entirely for the duration of the DLQ outage. A secondary infrastructure failure (DLQ topic) has cascaded into a full primary pipeline outage — precisely the head-of-line blocking scenario Share Groups were designed to prevent.

...

New Share Group Configuration Properties

New Broker Metrics

Share Group State Extension

...

1. Should the circuit breaker window counter be persisted in the share group state topic (for coordinator failover resilience), or is an in-memory counter acceptable given that a failover naturally resets the window?

   Proposed: In-memory counter is acceptable for the initial implementation. A coordinator failover resets the window, which is a safe failure mode — the circuit breaker may take up to one additional window period to re-trip after failover. Persisting the counter to the state topic adds complexity and write amplification that is not justified for a rolling-window metric. This can be revisited in a follow-on KIP if operational experience shows failover-induced false negatives are a problem.

2. Should auto.resume.enable default to true for development environments?

   Proposed: No — auto.resume.enable defaults to false in all environments. Differentiating defaults by environment would require broker-side environment detection, which is out of scope. Operators who want auto-resume behavior in development can set it explicitly. Defaulting to manual resume is the safer posture for production and avoids a footgun where a misconfigured production broker auto-resumes into a still-broken downstream service.

3. Should the pause-reason field be added to the existing DescribeGroups RPC, or introduced as a new DescribeShareGroupState RPC?

4. Should timed-out DLQ writes (ARCHIVED fall-through) count toward the rate threshold circuit breaker, or be tracked as a separate trigger condition only?

   Proposed: No. Timeout events and rate-threshold events are distinct failure modes with different operational meanings — a timeout indicates DLQ infrastructure unavailability, while a high DLQ rate indicates downstream consumer failure. Conflating them would make the circuit breaker harder to reason about and could cause false trips during DLQ infrastructure recovery. Timeout events are tracked separately via the DlqWriteTimeouts metric and should trigger operator alerting independently.