• Created by Unknown User (bbejeck) on Sep 17, 2024 2 minute read

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

Version 1 Next »

Status

Current state: Under Discussion 

Discussion thread: here 

JIRA: here 

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

Kafka Streams offers a comprehensive set of metrics for analyzing an application.  Although it's possible to deduce the health status of a Kafka Streams application from these metrics, operators require explicit status information metrics for better insights into the application's health.  In particular, operators will want to know the current state (CREATED, STARTING, PARTITIONS_REVOKED, PARTITIONS_ASSIGNED, RUNNING, PENDING_SHUTDOWN, DEAD) of each StreamThread in the client instance, the state (CREATED, REBALANCING, RUNNING, PENDING_SHUTDOWN, NOT_RUNNING, PENDING_ERROR, ERROR) of the client instance itself. 

Additionally, Kafka Streams provides several metrics with fine-grained details. Since recording all these metrics would result in a performance hit, Kafka Streams does not record all metrics out of the box.  There are different recording levels: info , debug , and trace set by the configuration metrics.recording.level . To avoid operator confusion or frustration regarding what metrics to expect, being able to observe the current recording level quickly is essential.

This KIP introduces new metrics to support Kafka Streams operators in quickly assessing an application's current health.

Public Interfaces

New Metrics

Each added metric will be on the client-level  and have the following tag:

  • type = client-metrics

The metrics for thread state will also contain the following tag:

  • thread-id=[thread-id] 
NameRecording LevelMetric TypeDescription
client-stateINFOGaugeThe current state of the Kafka Streams instance 
thread-stateINFOGaugeThe current state of the StreamThread 
recording-levelINFOGaugeThe level of metrics recording

Proposed Changes

These new metrics will be available at the INFO level.  Both the client-state  and thread-state will be updated to reflect the changes in the client or thread state.  The recording-level is determined on startup and contains the configured metrics recording level. 

Compatibility, Deprecation, and Migration Plan

  • No impact is expected for existing users, as no existing interfaces or behavior is being changed.
  • The expected impact of representing the state or recording-level will be neglible.

Test Plan

Existing metrics tests will be updated to support these new metrics.

Rejected Alternatives

N/A

  • No labels