Status

Current state: Under Discussion

Discussion thread:

JIRA: KAFKA-8833 

Motivation

In many use cases consumers are expected to fetch at the tail end of the log.

Public Interfaces

• Metrics : The consumer lag metrics will be added to the existing metrics interface in the broker 

Proposed Changes

The proposal is to have stats on consumer lag in terms for bytes as well as time. We will add broker level metrics aggregated across all partitions. Both metrics will be histogram.
 

• FetchLagTimeInMs: Histogram that will measure fetch log lag using the timestamp of the messages being fetched  
• FetchLagBytes: Histogram that will measure the fetch log lag by calculating the byte lag of the message being fetched

Compatibility, Deprecation, and Migration Plan

• No expected behavior change. The fetch code path will me used for measuring the consumer lag. The current fetch path is O(lg N), where N = number of segments. Since the consumer lag has to me measured in bytes this will have to change to an O(N) over the map that preserves the metadata of the segments at the tail end. Since the majority of the scenarios have a tail end fetch the impact is expected to be minimal.

Rejected Alternatives

If there are alternative ways of accomplishing the same thing, what were they? The purpose of this section is to motivate why the design is the way it is and not some other way.