• Created by Unknown User (bbejeck), last updated on Oct 01, 2024 2 minute read

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 2 Next »

Status

Current state: Under Discussion 

Discussion thread: here 

JIRA: here 

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

Kafka Streams offers a comprehensive set of metrics for analyzing an application.  Although it's possible to deduce the health status of a Kafka Streams application from these metrics, operators require explicit status information metrics for better insights into the application's health.  In particular, operators will want to know the current state (CREATED, STARTING, PARTITIONS_REVOKED, PARTITIONS_ASSIGNED, RUNNING, PENDING_SHUTDOWN, DEAD) of each StreamThread in the client instance, the state (CREATED, REBALANCING, RUNNING, PENDING_SHUTDOWN, NOT_RUNNING, PENDING_ERROR, ERROR) of the client instance itself.  While KIP-1076 extended the use of KIP-714 to provide additional application metrics, those metrics must map to either a Gauge (up/down counter) or a Sum (counter).  The existing Kafka Streams client-level metric is a String, so it can't be collected via the KIP-1076 extension.  To that end, we'll introduce a new numeric metric for the client-level state and take the opportunity to add a similar numeric metric for the thread-level state.  Since the thread-level state will also be of interest to users leveraging JMX metrics, we'll also add a thread-level metric with a String value available only via JMX. 

Additionally, Kafka Streams provides several metrics with fine-grained details. Since recording all these metrics would result in a performance hit, Kafka Streams does not record all metrics out of the box.  There are different recording levels: info , debug , and trace set by the configuration metrics.recording.level. To avoid operator confusion or frustration regarding what metrics to expect, being able to observe the current recording level quickly is essential.

This KIP introduces new metrics to support Kafka Streams operators in quickly assessing an application's current health.

Public Interfaces

New Metrics

Each added metric will be on the client-level  and have the following tag:

  • type = client-metrics

The metrics for thread state will also contain an additional tag:

  • thread-id=[thread-id] 
NameRecording LevelMetric TypeDescription
client-stateINFOGaugeThe current state of the Kafka Streams instance 
thread-stateINFOGaugeThe current state of the StreamThread 
recording-levelINFOGaugeThe level of metrics recording

Proposed Changes

These new metrics will be available at the INFO level.  Both the client-state  and thread-state will be updated to reflect the changes in the client or thread state.  The recording-level is determined on startup and contains the configured metrics recording level. 

Compatibility, Deprecation, and Migration Plan

  • No impact is expected for existing users, as no existing interfaces or behavior is being changed.
  • The expected impact of representing the state or recording-level will be negligible.

Test Plan

Existing metrics tests will be updated to support these new metrics.

Rejected Alternatives

N/A

  • No labels