Status

Current state: Under Discussion

Discussion thread: here

JIRA: here

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

Kafka Streams offers a comprehensive set of metrics for analyzing an application. Although it's possible to deduce the health status of a Kafka Streams application from these metrics, operators require explicit status information metrics for better insights into the application's health. In particular, operators will want to know the current state (CREATED, STARTING, PARTITIONS_REVOKED, PARTITIONS_ASSIGNED, RUNNING, PENDING_SHUTDOWN, DEAD) of each StreamThread in the client instance, the state (CREATED, REBALANCING, RUNNING, PENDING_SHUTDOWN, NOT_RUNNING, PENDING_ERROR, ERROR) of the client instance itself.

Additionally, Kafka Streams provides several metrics with fine-grained details. Since recording all these metrics would result in a performance hit, Kafka Streams does not record all metrics out of the box. There are different recording levels: info , debug , and trace set by the configuration metrics.recording.level . To avoid operator confusion or frustration regarding what metrics to expect, being able to observe the current recording level quickly is essential.

This KIP introduces new metrics to support Kafka Streams operators in quickly assessing an application's current health.

Public Interfaces

New Metrics

Each added metric will be on the client-level and have the following tag:

type = client-metrics

The metrics for thread state will also contain the following tag:

thread-id=[thread-id]

Name	Recording Level	Metric Type	Description
client-state	INFO	Gauge	The current state of the Kafka Streams instance
thread-state	INFO	Gauge	The current state of the `StreamThread`
recording-level	INFO	Gauge	The level of metrics recording

Proposed Changes

These new metrics will be available at the INFO level. Both the client-state and thread-state will be updated to reflect the changes in the client or thread state. The recording-level is determined on startup and contains the configured metrics recording level.

Compatibility, Deprecation, and Migration Plan

No impact is expected for existing users, as no existing interfaces or behavior is being changed.
The expected impact of representing the state or recording-level will be neglible.

Test Plan

Existing metrics tests will be updated to support these new metrics.

Rejected Alternatives

N/A