This page is meant as a template for writing a KIP. To create a KIP choose Tools->Copy on this page and modify with your content and replace the heading with the next KIP number and a description of your issue. Replace anything in italics with your own description.

Status

Current state:  [One of " Under Discussion", "Accepted", "Rejected"]

Discussion thread: here [Change the link from the KIP proposal email archive to your own email thread]
JIRA: here [Change the link from KAFKA-1 to your own ticket] TODO 

JIRA: TODO 

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

Describe the problems you are trying to solve.

Public Interfaces

Briefly list any new interfaces that will be introduced as part of this proposal or any existing interfaces that will be removed or changed. The purpose of this section is to concisely call out the public contract that will come along with this feature.

A public interface is any change to the following:

• Binary log format

• The network protocol and api behavior

• Any class in the public packages under clientsConfiguration, especially client configuration

  • org/apache/kafka/common/serialization

  • org/apache/kafka/common

  • org/apache/kafka/common/errors

  • org/apache/kafka/clients/producer

  • org/apache/kafka/clients/consumer (eventually, once stable)

• Monitoring

• Command line tools and arguments

• Anything else that will likely break existing users in some way when they upgrade

Proposed Changes

Describe the new thing you want to do in appropriate detail. This may be fairly extensive and have large subsections of its own. Or it may be a few sentences. Use judgement based on the scope of the change.

Compatibility, Deprecation, and Migration Plan

• What impact (if any) will there be on existing users?
• If we are changing behavior how will we phase out the older behavior?
• If we need special migration tools, describe them here.
• When will we remove the existing behavior?

Test Plan

Describe in few sentences how the KIP will be tested. We are mostly interested in system tests (since unit-tests are specific to implementation details). How will we know that the implementation works as expected? How will we know nothing broke?

Rejected Alternatives

KIP-714 introduced the ClientTelemetryReceiver interface to enable MetricsReporter instances running on brokers to collect client telemetry metrics. Typically a reporter stores metrics it receives in a "metrics registry" and exposes them in a format suitable for the external monitoring system is built for. This is exactly what JmxReporter does. Kafka clients and brokers explicitly delete their metrics when they are not used anymore. However for client telemetry metrics there isn't an explicit deletion mechanism.

Clients send their telemetry metrics at a regular interval, configured via telemetry subscriptions. Reporters are not aware of this push interval. Without the interval and an explicit deletion mechanism, reporters don't know when to clear client telemetry metrics and stop exposing them. This makes it hard to not expose stale metrics. Having uncertainty in metrics is a major problem as operators rely on them to manage their clusters.

Public Interfaces

• A new ClientTelemetryContext interface:

package org.apache.kafka.server.telemetry;

import org.apache.kafka.server.authorizer.AuthorizableRequestContext;


public interface ClientTelemetryContext extends AuthorizableRequestContext {

    int pushInterval();

    AuthorizableRequestContext authorizableRequestContext();
}

• A new method in ClientTelemetryReceiver

  Code Block
  language java
  /** * Called by the broker when a client reports telemetry metrics. The associated telemetry context * can be used by the metrics plugin to retrieve additional client information such as client ids, * endpoints or the push interval. * <p> * This method may be called from the request handling thread, and as such should avoid blocking. * * @param context the client telemetry context for the corresponding {@code PushTelemetryRequest} * api call. * @param payload the encoded telemetry payload as sent by the client. */ default void exportMetrics(ClientTelemetryContext context, ClientTelemetryPayload payload) { exportMetrics(context.authorizableRequestContext(), payload); }

  
  

Proposed Changes

• Deprecate the existing ClientTelemetryReceiver.exportMetrics() method

  Code Block
  language java
  /** * Called by the broker when a client reports telemetry metrics. The associated request context * can be used by the metrics plugin to retrieve additional client information such as client ids * or endpoints. * <p> * This method may be called from the request handling thread, and as such should avoid blocking. * <p> * This method is deprecated, {@link #exportMetrics(ClientTelemetryContext, ClientTelemetryPayload)} should be used * instead. * * @param context the client request context for the corresponding {@code PushTelemetryRequest} * api call. * @param payload the encoded telemetry payload as sent by the client. */ @Deprecated(since = "4.2.0") void exportMetrics(AuthorizableRequestContext context, ClientTelemetryPayload payload);

  
  

• The new ClientTelemetryReceiver.exportMetrics() has a default implementation that redirects to the now deprecated method. 

Compatibility, Deprecation, and Migration Plan

The new method has a default implementation so it should not break existing classes implementing ClientTelemetryReceiver.

Implementations that want to benefit from the new context need to override both the new and old exportMetrics() methods. The body for the old method can be the following for example, since it won't be called if void exportMetrics(ClientTelemetryContext context, ClientTelemetryPayload payload) is also implemented.

public void exportMetrics(AuthorizableRequestContext context, ClientTelemetryPayload payload) {
    throw new IllegalStateException("Should not be called");
}

Test Plan

Adequate unit and integration tests will be added to ensure ClientTelemetryReceiver implementations use the old or new contexts.

Rejected Alternatives

N/AIf there are alternative ways of accomplishing the same thing, what were they? The purpose of this section is to motivate why the design is the way it is and not some other way.