Status

Current state: "Under Discussion"

Discussion thread: here

JIRA: here

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

Kafka currently emits metrics for producers and consumers with different naming conventions in the topic metrics. Specifically, Kafka consumer metrics replace periods (.) in topic names with underscores (_), while Kafka producer metrics retain the period separator. This inconsistency can lead to confusion, misinterpretation of metric names, and additional overhead for users managing Kafka metrics.

This Kafka Improvement Proposal (KIP) aims to unify the naming conventions for Kafka consumer metrics by following the Kafka producer metrics nomenclature, preserving the original format of topic names (including periods).

The current inconsistency in metric naming across Kafka consumers and producers presents challenges for users monitoring and managing Kafka metrics. For example:

• Uniformity: By standardizing topic metric naming, users can streamline metric ingestion and analysis, ensuring a consistent naming convention for Kafka metrics.
• Reduce Complexity: Metrics tools and dashboards can use the same naming conventions, simplifying the setup and maintenance of monitoring systems.
• Enhance User Experience: Aligning consumer metrics with the producer naming convention reduces the cognitive load for users and makes metrics easier to interpret.

Currently, when a Kafka consumer metric is emitted, any periods (.) in the topic name are replaced with underscores (_). This behavior diverges from the Kafka producer metrics, which retain periods in topic names.

Public Interfaces

Affected Metrics

The following Kafka consumer metrics will be impacted by this change:

• consumer-fetch-manager-metrics
  • Topic:
    • bytes-consumed-total
    • bytes-consumed-rate
    • records-consumed-rate
    • fetch-size-max
    • records-consumed-total
    • fetch-size-avg
    • records-per-request-avg
  • Topic-Partition:
    • records-lag-max
    • records-lag
    • records-lag-avg
    • preferred-read-replica
    • records-lead-min
    • records-lead-avg
    • records-lead

With this KIP, these metrics will emit topic names without any modification to periods, matching the producer naming scheme. Exisiting consumer-fetch-manager-metrics will be marked deprecated and shall be removed in future releases.

Proposed Changes

• Remove Replacement of Periods in Consumer Metrics Topic Names:

  • Modify the Kafka consumer metrics to preserve the topic names as they are, without replacing periods with underscores.
  • This change will align Kafka consumer metrics with the Kafka producer metrics naming convention.

Compatibility, Deprecation, and Migration Plan

To ease the transition for users who rely on the existing underscore-based consumer metric naming:

• Users relying on the existing underscore-based topic names for consumer metrics do not neet to immediately change anything as old metrics will be only marked as deprectaed.
• However once removed in future releases then users may need to update their monitoring systems and dashboards to use the original topic names (with periods).

Test Plan

• Unit Tests: Update unit tests for Kafka consumer metrics to validate that topic names with periods are emitted without modification.

Rejected Alternatives

N/A