...

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

KIP-932 introduced share groups to allow multiple share consumers to consume messages from the same partition concurrently. While this adds flexibility, it also introduces challenges in observability. Currently, there is no visibility into consumption progress at the granularity of individual share partitions, making it difficult to detect imbalances, identify slow share consumers, or troubleshoot performance issues.

Introducing share partition lag provides fine-grained visibility into consumption progress, enabling operators to monitor share group behavior more effectively and make informed operational decisions. It also enables future automation opportunities—external autoscalers such as KEDA could use this lag to dynamically scale share consumers based on real-time workload demand, improving efficiency and reducing manual intervention.

Looking ahead, the plan is to build on this foundation by introducing an assignor that can allocate share group members to partitions based on their partition-level backlogs, ensuring more balanced load distribution and improved overall consumption efficiency.

Proposed Changes

Share Partition Lag Computation

...

• AVAILABLE

• ACQUIRED

• ACKNOWLEDGED

• ARCHIVED

To measure the lag, we first need to determine the highest offset in the underlying partition log, which defines the upper boundary of records currently available for consumption.

...

For read-committed consumers, the Log Stable Offset (LSO) is used, as it marks the end of all committed data visible to consumers.

...

Similar to regular consumer groups, this offset will be retrieved using the read-uncommitted isolation level. Consequently, the Log End Offset (LEO) will be used as the reference point for measuring lag, as it represents the latest offset in the partition, including both committed and uncommitted records.

Based on this, the lag for a share partition is defined as:

...

For example, consider the following topic - partition:

+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+
|   0   |   1   |   2   |   3   |   4   |   5   |   6   |   7   |   8   |   9   |   10  | <- offset
| Archv | Archv | Acqrd | Avail | Acqrd | Acked | Archv | Avail | Avail | Avail | Avail | <- state
|       |       |   1   |   2   |   1   |       |       |       |       |       |       | <- delivery count
+-------+-------+---^---+-------+-------+-------+-------+-------+-------+---^---+-------+
                    |                                                       |
                    +-- Share-partition start offset (SPSO)                 +-- Share-partition end offset (SPEO)

...

To make this information available to users, the Group Coordinator retrieves it through the ReadShareGroupStateSummary API and computes the share-partition lag, which is then included in the response to the DescribeShareGroupOffsets request invoked by Admin.listShareGroupOffsets(). To calculate the lag, the Group Coordinator issues an Admin.listOffsets() call to fetch the end offset of the underlying partition and then applies the share-partition lag formula defined above to derive the lag value.Looking ahead, the plan is to implement an assignor that allocates members to partitions based on partition-level backlogs.

Public Interfaces

Client API changes

...