Status

Current state: Under Discussion Voting in progress

Discussion thread: https://lists.apache.org/thread/z4o5093zqxmc18nzb74vyhvb1hb399v4

Vote thread: https://lists.apache.org/thread/jxf789v0kdpvsb9wsosghh57odkksxt7

JIRA:

...

• Backpressure: When an internal record buffer for a partition exceeds its capacity, Kafka Streams pauses that partition to prevent further fetching (StreamTask. addRecords()). It resumes the partition once the buffer drains below the threshold (StreamTask.resumePollingForPartitionsWithAvailableSpace()).
• Rebalance lifecycle: After a rebalance, Kafka Streams pauses all partitions that are not yet owned by fully initialized tasks (TaskManager). Partitions are only resumed after state store restoration completes.
• Changelog restore management: The StoreChangelogReader  pauses  pauses and resumes changelog partitions on the restore consumer depending on whether the corresponding tasks still exist.

...

1. How many partitions are currently paused? A number of paused partitions may indicate backpressure issues, slow state restoration, or application-level processing bottlenecks.
2. Which specific partitions are paused? Knowing which partitions are paused helps pinpoint which topics or partitions are experiencing issues.
3. How long has a partition been paused? A partition that has been paused for an extended period may indicate a stuck consumer, an unrecoverable error in processing, or a state store restoration that is taking too long.
4. How frequently are partitions being paused? A high rate of pause calls may indicate instability in the application's flow control logic or repeated rebalances triggering pause/resume cycles.

This KIP proposes adding new consumer metrics that track paused partitions and their pause duration.

...

consumer-fetch-manager-metrics

...

Note: These are per-partition metrics. Operators should be aware that registering metrics per partition increases cardinality , consistent with existing per-partition consumer metrics such as records-lag  and records-lead . Cardinality increases linearly with the number of assigned partitions. In environments with large partition counts, this , which may increase memory usage in the metrics registry and in downstream monitoring systems. The paused-partitions-count aggregate metric is available at INFO level for general monitoring, while the per-partition metrics are intended for detailed debugging when partition-level visibility is needed.

Proposed Changes

Track Pause Timestamp

Currently, each partition's internal state tracks whether it is paused with a simple boolean flag. To support the paused-partitions-duration-seconds  metric, this KIP adds a timestamp that records when the partition was paused. The timestamp is set when pause()  is called, cleared (reset to -1) when resume() is called, and also reset to -1 cleared on partition reassignment regardless of prior pause status. This ensures that a partition re-assigned to the same consumer after a rebalance starts with a clean pause state.

consumer-coordinator-metrics

A paused-partitions-count  gauge is registered in the consumer's top-level metrics / paused-partitions-duration-seconds-max  gauges and paused-partitions-rate / paused-partitions-total sensors are registered at coordinator level. When queried, this gauge computes these gauges compute the current count of paused partitions and max paused duration by reading from the consumer's subscription state. Since the consumer already maintains the set of paused partitions internally, no additional data structures are needed for this metricSensor metrics are updated on each pause() call, incrementing the cumulative count and contributing to the rate calculation.

consumer-fetch-manager-metrics

The paused-partitionspartition  and / paused-partitionspartition-duration-seconds  gauges  gauges and paused-partition-rate / paused-partition-total sensors are registered per partition in the fetch manager metrics, alongside the existing per-partition lag and lead metrics. They follow the same lifecycle:

• On partition assignment: The metrics are registered for each newly assigned partition.
• On partition revocation: The metrics are removed for each revoked partition, preventing stale metrics from accumulating. Both Gauge metrics are implemented as on-demand gauges that read directly from the partition's pause state when queried, rather than being updated on every pause/resume call.

...

• Verify that the pause timestamp is set when pause()  is called, cleared on resume() , and reset on partition reassignment.
• Verify that the paused-partitions-count  metric is coordinator-level metrics are registered and returns the correct count as partitions are paused and resumed.
• Verify that paused-partitions  and paused-partitions-duration-seconds  per-partition metrics are registered per partition, return correct values, and are removed when partitions are revoked.

...