Status

Current state: Under Discussion

Discussion thread: here

JIRA:

Motivation

All Kafka component register AppInfo metrics to track the application start time or commit-id etc. These metrics are useful for monitoring and debugging. However, the AppInfo doesn't provide client-id, which is an important information for custom metrics reporter.

The AppInfoParser class registers a JMX MBean with the provided client-id, but when it adds metrics to the Metrics registry, the client-id is not included. This KIP aims to add the client-id as a tag.

Public Interfaces

Currently MetricName:

[name=start-time-ms, group=app-info, description=Metric indicating start-time-ms, tags={}]
[name=commit-id, group=app-info, description=Metric indicating commit-id, tags={}]
[name=version, group=app-info, description=Metric indicating version, tags={}]

New MetricName:

[name=start-time-ms, group=app-info, description=Metric indicating start-time-ms, tags={client-id=<component-id>}]
[name=commit-id, group=app-info, description=Metric indicating commit-id, tags={client-id=<component-id>}]
[name=version, group=app-info, description=Metric indicating version, tags={client-id=<component-id>}]

Proposed Changes

1) The new MBean will include a new tag, client-id

private static void registerMetrics(Metrics metrics, AppInfo appInfo, String clientId) {
    if (metrics != null && clientId != null) {
        metrics.addMetric(metricName(metrics, "version", Map.of("client-id", clientId)), new ImmutableValue<>(appInfo.getVersion()));
        metrics.addMetric(metricName(metrics, "commit-id", Map.of("client-id", clientId)), new ImmutableValue<>(appInfo.getCommitId()));
        metrics.addMetric(metricName(metrics, "start-time-ms", Map.of("client-id", clientId)), new ImmutableValue<>(appInfo.getStartTimeMs()));
    } else if (metrics != null) {
        metrics.addMetric(metricName(metrics, "version", Map.of()), new ImmutableValue<>(appInfo.getVersion()));
        metrics.addMetric(metricName(metrics, "commit-id", Map.of()), new ImmutableValue<>(appInfo.getCommitId()));
        metrics.addMetric(metricName(metrics, "start-time-ms", Map.of()), new ImmutableValue<>(appInfo.getStartTimeMs()));
    }
}

2) When AppInfoParser registers an MBean, it will register both the new and the deprecated MBean.

public static synchronized void registerAppInfo(String prefix, String id, Metrics metrics, long nowMs) {
    try {
        // skip...

        registerMetrics(metrics, mBean, null); // prefix will be added later by JmxReporter
        registerMetrics(metrics, mBean, id);
    } catch (JMException e) {
        log.warn("Error registering AppInfo mbean", e);
    }
}

3) When unregisters an MBean, also remove the new and deprecated MBean.

public static synchronized void unregisterAppInfo(String prefix, String id, Metrics metrics) {
    MBeanServer server = ManagementFactory.getPlatformMBeanServer();
    try {
        ObjectName name = new ObjectName(prefix + ":type=app-info,id=" + Sanitizer.jmxSanitize(id));
        if (server.isRegistered(name))
            server.unregisterMBean(name);

        unregisterMetrics(metrics, null);
        unregisterMetrics(metrics, id);
    } catch (JMException e) {
        log.warn("Error unregistering AppInfo mbean", e);
    } finally {
        log.info("App info {} for {} unregistered", prefix, id);
    }
}


private static void unregisterMetrics(Metrics metrics, String clientId) {
    if (metrics != null && clientId != null) {
        metrics.removeMetric(metricName(metrics, "version", Map.of("client-id", clientId)));
        metrics.removeMetric(metricName(metrics, "commit-id", Map.of("client-id", clientId)));
        metrics.removeMetric(metricName(metrics, "start-time-ms", Map.of("client-id", clientId)));
    } else if (metrics != null) {
        metrics.removeMetric(metricName(metrics, "version", Map.of()));
        metrics.removeMetric(metricName(metrics, "commit-id", Map.of()));
        metrics.removeMetric(metricName(metrics, "start-time-ms", Map.of()));
    }
}

Compatibility, Deprecation, and Migration Plan

For users currently relying on the AppInfoMBean without client-id, it will remain available until Kafka 5.0. New users are encouraged to use the new AppInfoMBean, which includes the client-id tag. This deprecation will be documented in upgrade.html.

In Kafka 5.0, the deprecated AppInfoMBean will be removed.

Test Plan

change unit test or Integration test to verify the new method.

Rejected Alternatives

A new deprecated configuration could be introduced to control the registration of this MBean.

Nevertheless, this approach is considered unnecessary, as it would introduce complexity without significant benefit.