Status

Current state: Under Discussion

Discussion thread: here

JIRA: here

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

Currently, there are two metrics for measuring the traffic in topic dimensions: MessagesInPerSec, BytesInPerSec, but there are two problems:
1. It is difficult to intuitively reflect the problem of topic partition traffic inclination through these indicators, and it is impossible to clearly see which partition has the largest traffic and the traffic situation of each partition. But the partition dimension can solve this.
2. For the sudden increase in topic traffic (for example, this will lead some followers to out of Isr, which can be avoided by appropriately increasing the number of partitions.), the metrics of the partition dimension can help to provide guidance on whether to expand the partition.

On the whole, I think it is very meaningful to add traffic metrics of partition dimension, especially the issue of traffic skew.

Public Interfaces

• add a topic-partition dimension metric for per broker: kafka.server:type=BrokerTopicPartitionMetrics,name=PartitionBytesInPerSec
  • description: Used to describe client's bytesInPerSec traffic for a topic-partition.
  • tags: topic and partition, e.g. kafka.server:type=BrokerTopicPartitionMetrics,name=PartitionBytesInPerSec,topic=test1,partition=0
• add a topic-partition dimension metric for per broker: kafka.server:type=BrokerTopicPartitionMetrics,name=PartitionMessagesInPerSec
  • description: Used to describe client's MessagesInPerSec traffic for a topic-partition.
  • tags: topic and partition, e.g. kafka.server:type=BrokerTopicPartitionMetrics,name=PartitionMessagesInPerSec,topic=test1,partition=0

Proposed Changes

Referring to the "Public Interfaces" section, the metrics of the two partition dimensions "PartitionMessagesInPerSec" and "PartitionBytesInPerSec" are added, which will be updated when the client writes to the log.

Compatibility, Deprecation, and Migration Plan

The new metric added to this kip will not have any impact on the existing behavior.

Test Plan

Add related unit tests in MetricsTest.scala.

Rejected Alternatives