Versions Compared

Old Version 10

changes.mady.by.user Ken Huang

Saved on

compared with

New Version Current

changes.mady.by.user Ken Huang

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Table of Contents

Status

Current state: "Under DiscussionAccepted"

Discussion thread: here

Vote thread: here

JIRA: KAFKA-17876 

Motivation

The metrics were originally named in the `kafka.<COMPONENT>` format but were unexpectedly altered when we moved Java classes from the `kafka.server` package to the `org.apache.kafka.server` package as part of [commit](3.7).

Subsequent developers didn’t notice this change and began using the new package name as the `metricName`. The correct naming should actually revert to `kafka.<COMPONENT>`.

...

Jira
serverASF JIRA
columnIdsissuekey,summary,issuetype,created,updated,duedate,assignee,reporter,customfield_12311032,customfield_12311037,customfield_12311022,customfield_12311027,priority,status,resolution
columnskey,summary,type,created,updated,due,assignee,reporter,Priority,Priority,Priority,Priority,priority,status,resolution
serverId5aa69414-a9e9-3523-82ec-879b028fb15b
keyKAFKA-17876
Jira
serverASF JIRA
columnIdsissuekey,summary,issuetype,created,updated,duedate,assignee,reporter,customfield_12311032,customfield_12311037,customfield_12311022,customfield_12311027,priority,status,resolution
columnskey,summary,type,created,updated,due,assignee,reporter,Priority,Priority,Priority,Priority,priority,status,resolution
serverId5aa69414-a9e9-3523-82ec-879b028fb15b
keyKAFKA-19150

Motivation

Kafka metrics have traditionally followed a consistent naming convention using the kafka.<COMPONENT> format. However, some inconsistencies were unintentionally introduced during recent refactorings:

  • The metrics under org.apache.kafka.server:type=AssignmentsManager were originally named using the kafka.<COMPONENT> format. This changed unexpectedly when classes were moved from the kafka.server package to org.apache.kafka.server as part of this commit.
  • Similarly, the org.apache.kafka.storage.internals.log:type=RemoteStorageThreadPool metric was initially intended to follow the same convention. However, an incorrect metric name was introduced during initialization in a later patch.

These changes went unnoticed, and subsequent users began using the new org.apache.kafka.<COMPONENT> pattern, diverging from the established convention.

This KIP proposes to standardize Kafka metric naming by reverting the following metrics to the correct kafka.<COMPONENT> format to ensure consistency across the system:

  • org.apache.kafka.server:type=AssignmentsManager → kafka.server:type=AssignmentsManager

  • org.apache.kafka.storage.internals.log:type=RemoteStorageThreadPool → kafka.log.remote:type=RemoteStorageThreadPool

This change will align these metrics with the broader Kafka metrics ecosystem and avoid confusion for users and tooling that rely on consistent naming.

Public Interface

  • org.apache.kafka.storage.internals.log.RemoteStorageThreadPool 
  • org.apache.kafka.server.log.remote.storage.RemoteStorageMetrics 
  • org.apache.kafka.server.AssignmentsManager

Proposed Changes

...

Introduce new metrics, consistent with other Kafka metrics.

This change will impact external monitoring systems that rely on the old metric name; corresponding updates will be required.

Deprecate the following metrics

  • org.apache.kafka.storage.internals.log:type=RemoteStorageThreadPool.RemoteLogReaderTaskQueueSize
  • org.apache.kafka.storage.internals.log:type=RemoteStorageThreadPool.RemoteLogReaderAvgIdlePercent
  • org.apache.kafka.server:type=

...

  • AssignmentsManager.QueuedReplicaToDirAssignments

Introduce following new metric: 

  • kafka.log.remote:type=RemoteStorageThreadPool.RemoteLogReaderAvgIdlePercent
  • kafka.log.remote:type=RemoteStorageThreadPool.RemoteLogReaderTaskQueueSize
  • kafka.server:type=

...

  • AssignmentsManager.QueuedReplicaToDirAssignments

This change will impact external monitoring systems that rely on this metric, and corresponding modifications will be needed to support the new name.the old metric name; corresponding updates will be required.

Due to 

Compatibility, Deprecation, and Migration Plan

...

  1. We will add a deprecated annotation on following constant
    1. `RemoteStorageMetrics#REMOTE_LOG_READER_TASK_QUEUE_SIZE_METRIC`
    2. `RemoteStorageMetrics#REMOTE_LOG_READER_AVG_IDLE_PERCENT_METRIC`
    3. `AssignmentsManager#QUEUED_REPLICA_TO_DIR_ASSIGNMENTS_METRIC`
  2. Update the Kafka documentation to note MBean will be replaced
    1. `org.apache.kafka.storage.internals.log:type=RemoteStorageThreadPool.RemoteLogReaderTaskQueueSize`
    2. `org
    Add new warn log which is "org
    1. .apache.kafka.storage.internals.
    server
    1. log:type=
    AssignmentsManager will be deprecated in the future, it will rename to
    1. RemoteStorageThreadPool.RemoteLogReaderAvgIdlePercent`
    2. `org.apache.kafka.server:type=AssignmentsManager
    "in `AssignmentsManager#metricName` method.In the next major release, we will update `AssignmentsManager#metricName` method to return
    1. .QueuedReplicaToDirAssignments`
  3. Register folowing new metrics
    1. `kafka.log.remote:type=RemoteStorageThreadPool.RemoteLogReaderTaskQueueSize`
    2. `kafka.log.remote:type=RemoteStorageThreadPool.RemoteLogReaderAvgIdlePercent`
    3. `kafka.server:type=
    AssignmentsManager`.

Test Plan

    1. AssignmentsManager.QueuedReplicaToDirAssignments`
  2. Delete following all usage in code base at the Kafka 5.0
    1. `RemoteStorageMetrics#REMOTE_LOG_READER_TASK_QUEUE_SIZE_METRIC`
    2. `RemoteStorageMetrics#REMOTE_LOG_READER_AVG_IDLE_PERCENT_METRIC`
    3. `AssignmentsManager#QUEUED_REPLICA_TO_DIR_ASSIGNMENTS_METRIC`

Test Plan

  • Add new tests to verify the new metric is correctly registered and behaves as expected.
  • Ensure backward compatibility by verifying deprecated metric behavior until removal.

...

Rejected Alternatives

n/a