Status

Current state: Under Discussion

Discussion thread: here

JIRA: KAFKA-3715 Higher granularity streams metrics

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

• This KIP proposes the addition of a RecordLevel in Sensor.java. This associates each sensor with a record level and the metrics and computations associated with the sensor are recorded or calculated only if the metric config of the client requires these metrics to be recorded. For example, a sensor associated with a RecordLevel.DEBUG record level will only record metrics, (and correspondingly compute the time.milliseconds or nanoseconds if required by its metrics), if the client config for metric record levels is set to DEBUG. This provides a mechanism for clients to toggle metric granularity without a change to individual sensors, and we do not perform unnecessary intensive computations associated on metrics which must not be recorded.

Public Interfaces

• Changes to public class Sensor.java to include Sensor.RecordLevel
• addition of static string in CommonClientConfigs.java

Proposed Changes

• The constructor of Sensor.java now takes a Sensor.RecordLevel as a parameter. This is an enum which has DEBUG and INFO record levels, and can be extended in future to include other levels.

Sensor(Metrics registry, String name, Sensor[] parents, MetricConfig config, Time    time, long  inactiveSensorExpirationTimeSeconds, RecordLevel  recordLevel)

• The ClientConfig will have a new option called a METRICS_RECORD_LEVEL_CONFIG which is set to match the record levels of INFO or DEBUG while defining the consumer config

• The shouldRecord() method is used to compare the value of metric record level in the consumer config and the RecordLevel associated with the sensor, to determine if metrics should be computed and recorded.

Compatibility, Deprecation, and Migration Plan

• no migration plan needed, backward compatible constructor for Sensor.java which adds RecordLevel.INFO where a record level is not specified.

Rejected Alternatives

• We considered whether we want to register a DEBUG sensor at all if the config is INFO. Currently we do register it, but we do not record any values. This was rejected because it would add complexity to Sensor.java. A client could attempt to register a metric and report a metric against a sensor which is not internally registered.