Status

Current state: "Under Discussion" Accepted

Discussion thread: here

Vote Discussion thread: here

JIRA: here

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

...

With this KIP, these metrics will also emit topic names without any modification to periods, matching the producer naming scheme. Exisiting New consumer-fetch-manager-metrics will not have replaced dot (.) in topic names. "However for backward compatibility, the period replaced metrics will also be emitted. They are now deprecated and will be removed in a future major Kafka version (5.0)."

Proposed Changes

• Remove Replacement of Periods Add additional metric without periods replacement in Consumer Metrics Topic Names:

  • Modify If topic contains period (.), add metrics with same name in the Kafka consumer metrics to preserve the topic names as they are, without replacing periods with underscores.
  • This change will align Kafka consumer metrics with the Kafka producer metrics naming convention.

Compatibility, Deprecation, and Migration Plan

N/A

Test Plan

• Unit Tests: Update unit tests for Kafka consumer metrics to validate that topic names with periods are emitted without modification.

...

• • . Keep emitting the existing period replaced metrics as well for backward compatibility.
  • If topic doesn't contain period (.), no change to existing metrics. 

Note: A drawback of the approach is that metrics for topics containing periods will be emitted twice: once with the actual topic name and once with periods replaced. Users must monitor either metric and may need to discard duplicate metrics for accurate reporting. But this is necessary to maintain backward compatibility and prevent disruption to existing user reporting.

Example:

Kafka 4.x:

• Topic Name: topic.a.b
  • Deprecated Metric: A metric with the tag topic_a_b will continue to be emitted, but it will be marked as deprecated.
  • New Metric: A new metric with the topic tag topic.a.b and same name has been introduced. Users should transition to this new metric.
• Topic Name: topic_c_d
  • Metric: A metric with the tag topic_c_d will continue to be emitted without any changes.
  • No Additional Metrics: No additional metrics are needed as the existing metric functions correctly.

Kafka 5.0:

The period replacement policy will be completely removed.

Compatibility, Deprecation, and Migration Plan

To ease the transition for users who rely on the existing underscore-based consumer metric naming:

• Users relying on the existing underscore-based topic names for consumer metrics do not neet need to immediately change anything as old metrics will be only marked as deprectaed.
• However, to ensure accurate reporting, users may need to filter out duplicate metrics, as new metrics with actual topic names will be emitted alongside the deprecated ones.
• Once deprecated metrics once removed in future releases then users may need to update their monitoring systems and dashboards to use the original topic names (with periods).

Test Plan

• Unit Tests: Update unit tests for Kafka consumer metrics to validate that topic names with periods are also emitted without modification, only when required.

Rejected Alternatives

• Correct the topic names without deprecating existing metrics: Users who are currently monitoring metrics with replaced topic names might experience disruption.