Status

Current state: "Voting in progress"

Discussion thread: here

Voting thread: here

JIRA: here

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

As owner of Kafka cluster, but not the owner of clients, I am not capable of telling which topics/partitions are being actively read from (= have running clients that are sending fetch requests).

The existing metric kafka.server:type=BrokerTopicMetrics,name=TotalFetchRequestsPerSec,topic=topic aggregates both fetch requests made by clients and replicas.

So for partitions with replication factor > 1, even if the partition is no longer read from by clients, the above metric is still growing due to requests made by replicas.

Public Interfaces

The following new broker-side metric is proposed:

• kafka.server:type=BrokerTopicMetrics,name=ConsumerFetchRequestsPerSec
• kafka.server:type=BrokerTopicMetrics,name=ConsumerFetchRequestsPerSec,topic=mytopic (one per topic)

It would be similar to the already-existing TotalFetchRequestsPerSec metric.

Proposed Changes

1. Add new metric mentioned above
2. Update ConsumerFetchRequestsPerSec metric only when the fetch request is coming from client only (-1 != replicaId).

Compatibility, Deprecation, and Migration Plan

These are new metrics and there should be no impact on existing users.

Rejected Alternatives

1. Do not provide new metrics - does not satisfy the need in environment where cluster owner has no knowledge about clients.
2. Provide two metrics: ConsumerFetchRequestsPerSec and ReplicationFetchRequestsPerSec:
   1. benefits: user is capable of accessing the desired metric without the need to find the values of others (what might be beneficial to e.g. jmxterm users)
   2. drawbacks: increased footprint (extra metric per topic & need to update it per request)