Status

Current state: Under DiscussionAccepted

Discussion thread: TBD link

JIRA:

...

For the above purposes, we want to 1) cleanup Streams Built-in Metrics to have more out-of-the-box useful metrics while trimming those non-useful ones because the current APIs are not very intuitive from its naming to reason about its semantics (this proposal includes removing some redundant APIs as well as refactoring the parent-child metrics relationships, details below), and 2) improve APIs for User Customized Metrics that let users register them own metrics, based on its "operationName / scopeName / entityName" notions; we would simplify this interface for user's needs, plus making sure it functions correctly.

...

// deprecated APIs: use {@link Sensor#record(double)} directly instead.

@Deprecated
void recordLatency(final Sensor sensor, final long startNs, final long endNs);

@Deprecated
void recordThroughput(final Sensor sensor, void final long value);

//@Deprecated updated
* APIs@deprecated javadocs

  /*
   * Add a latency and rate sensor for a specific operation, which since 2.5. Use {@link addLatencyRateTotalSensor} instead
Sensor addLatencyAndThroughputSensor(... )

@Deprecated 
* @deprecated since 2.5. Use {@link addRateTotalSensor} instead
Sensor addThroughputSensor(... )

// updated APIs javadocs

  /*
   * Add a latency, rate and total sensor for a specific operation, which will include the following metrics:
   * <ol>
   * <li>average latency</li>
   * <li>max latency</li>
   * <li>invocation rate (num.operations / time unit)</li>
   * <li>total invocation count</li>
   * </ol>
   * Whenever a user record this sensor via {@link Sensor#record(double)} etc,
   * it will be counted as one invocation of the operation, and hence the rate / count metrics will be updated accordingly;
   * and the recorded latency value will be used to update the average / max latency as well. The time unit of the latency can be defined
   * by the user.
   *
   * Note that you can add more metrics to this sensor after created it, which can then be updated upon {@link Sensor#record(double)} calls;
   * but additional user-customized metrics will not be managed by {@link StreamsMetrics}.
   *
   * @param scopeName          name of the scope, which will be used as part of the metrics type, e.g.: "stream-[scope]-metrics".
   * @param entityName         name of the entity, which will be used as part of the metric tags, e.g.: "[scope]-id" = "[entity]".
   * @param operationName      name of the operation, which will be used as the name of the metric, e.g.: "[operation]-latency-avg".
   * @param recordingLevel     the recording level (e.g., INFO or DEBUG) for this sensor.
   * @param tags               additional tags of the sensor
   * @return The added sensor.
   */
  Sensor addLatencyAndRateSensoraddLatencyRateTotalSensor(final String scopeName,
                                   final String entityName,
                                   final String operationName,
                                   final Sensor.RecordingLevel recordingLevel,
                                   final String... tags);

  /*
   * Add a rate and a total sensor for a specific operation, which will include the following metrics:
   * <ol>
   * <li>invocation rate (num.operations / time unit)</li>
   * <li>total invocation count</li>
   * </ol>
   * Whenever a user record this sensor via {@link Sensor#record(double)} etc,
   * it will be counted as one invocation of the operation, and hence the rate / count metrics will be updated accordingly.
   *
   * Note that you can add more metrics to this sensor after created it, which can then be updated upon {@link Sensor#record(double)} calls;
   * but additional user-customized metrics will not be managed by {@link StreamsMetrics}.
   *
   * @param scopeName          name of the scope, which will be used as part of the metrics type, e.g.: "stream-[scope]-metrics".
   * @param entityName         name of the entity, which will be used as part of the metric tags, e.g.: "[scope]-id" = "[entity]".
   * @param operationName      name of the operation, which will be used as the name of the metric, e.g.: "[operation]-latency-avg".
   * @param recordingLevel     the recording level (e.g., INFO or DEBUG) for this sensor.
   * @param tags               additional tags of the sensor
   * @return The added sensor.
   */
  Sensor addRateSensoraddRateTotalSensor(final String scopeName,
                            final String entityName,
                            final String operationName,
                            final Sensor.RecordingLevel recordingLevel,
                            final String... tags);

Users can create a sensor via either `addLatencyAndRateSensor` or `addRateSensor`, which will be pre-registered with the latency / rate metrics already; more metrics can then be added to the returned sensors in addition to the pre-registered ones. When recording a value to the sensor, users should just use `Sensor#record()` directly on the sensor itself.

...

version | commit-id (static gauge)

application-id (static gauge)

topology-description (static gauge)

state (dynamic gauge)

alive-stream-threads (dynamic gauge)

process-latency (avg | max)

process (rate | total)

punctuate-latency (avg | max)

punctuate (rate | total)

commit-latency (avg | max)

commit (rate | total)

poll-latency (avg | max)

poll (rate | total)

process | 

punctuate | commit | 

active-task-process (ratiopoll-ratio (dynamic gauge)

standby-task-process (ratio)

dropped-recordscreated | closed (rate | total)

poll-

records (

avg | 

max)

process-recordsrecord-lateness (avg | max)

active-process-

ratio (

dynamic 

gauge)

suppression-buffer-size (avg | max)

suppression-buffer-count (avg | max)

standby-process-ratio (dynamic gauge)

dropped-records (rate | total)

active-buffer-count (dynamic gauge)

enforced-processingexpired-window-record-drop (rate | total)

record-lateness (avg | max)

suppression-emit (rate | total)

put | putsuppression-if-absent .. | get-latency (avg | maxbuffer-size (avg | max)

suppression-buffer-count (avg | max)

(put | put-if-absent .. | get)-latency (avg | max)

(put | put-if-absent .. | get) (rate)

hit-ratio (avg | min | max)

...

1. We will remove most of the parent sensors with `level-tag=all` except two casesone case.  The main idea is to let users to do rolling-ups themselves only if necessary so that we can save necessary metrics value aggregations. For those two these exceptional casescase, two one parent-child sensor relationship is maintained because it is a bit tricky for users to do the rolling up correctly.
2. We will keep all LEVEL-0 (instance) and LEVEL-1 (thread) sensors as INFO, and most of lower level sensors as DEBUG reporting level. They only exception is active/standby-task-process and dropped / skipp-records
   1. active/standby-task-process indicate the percentage that the current hosting thread is spending on processing them.
   2. dropped/skipped records indicate unexpected errors during processing and hence need to be paid attention by users. Their semantics though are a bit different: skipped records are those skipped at the very beginning of the process and hence not even traverse the topology at all; dropped-records are those dropped in the middle of the topology, and are not necessarily corresponding to a 1-1 mapping to the source records since one source records may be transformed to multiple intermediate records which are then dropped later.
3. For some metrics that are only useful for a specific type of entities, like "suppression-emit", we will only create the sensors lazily in order to save unnecessary costs for metrics reporters to iterate those empty sensors.
4. Some of the lower level metrics like "forward-rate" and "destroy-rate" are removed directly since they are overlapping with other existing metrics already. Here are a list of removed / replaced sensors:
   
   

late-records-drop: INFO at processor node level, replaced by INFO task-level "dropped-records".

skipped-records: INFO at thread and processor node level, replaced by INFO task-level "dropped-records".

expired-window-record-drop: DEBUG at state store level, replaced by INFO task-level "dropped-records".

forward-rate: DEBUG at processor-node level, replaced by DEBUG processor node level "process-rate".

destorydestroy-rate: DEBUG at taskprocessor-node level, covered by INFO thread-level "task-closed-rate".

create-rate: DEBUG at taskprocessor-node level, covered by INFO thread-level "task-create-rate". by INFO thread-level "task-create-rate".

Release History

2.4

• client-level metrics:
  • added:
    • version 
    • commit-id
    • application-id
    • topology-description
    • state

2.5

• thread-level metrics:
  • refactored:
    • process-latency (avg | max)
    • process (rate | total) 
    • punctuate-latency (avg | max)
    • punctuate (rate | total)
    • commit-latency (avg | max)
    • commit (rate | total)
    • poll-latency (avg | max)
    • poll (rate | total)
    • task-created | closed (rate | total)
  • removed:
    • skipped-records
• task-level metrics:
  • refactored:
    • process-latency (avg | max)
    • process (rate | total) 
    • punctuate-latency (avg | max)
    • punctuate (rate | total)
    • commit-latency (avg | max)
    • commit (rate | total)
    • enforced-processing (rate | total)
    • record-lateness (avg | max)
  • added:
    • dropped-records (rate | total)
  • removed:
    • expired-window-record-drop
• processor-node-level:
  • refactored:
    • process (rate | total)
    • suppression-emit (rate | total)
  • removed:
    • process-latency (avg | max)
    • late-records-drop
    • skipped-records
    • forward-rate
    • destroy-rate
    • create-rate
• state-store-level:
  • refactored:
    • suppression-buffer-size (avg | max)
    • suppression-buffer-count (avg | max)
    • (put | put-if-absent .. | get)-latency (avg | max)
    • (put | put-if-absent .. | get) (rate)
  • removed:
    • expired-window-record-drop
• cache-level:
  • refactored:
    • hit-ratio (avg | min | max)

2.6

• client-level metrics:
  • added: 
    • alive-stream-threads
• thread-level metrics:
  • added: 
    • process | punctuate | commit | poll-ratio
    • poll-records (avg | max)
    • process-records (avg | max)
• task-level metrics:
  • added: 
    • active-process-ratio

Proposed Changes

As above.

...

"built.in.metrics.version":


type: Enum 
values: {"0.10.0", "0.10.1", ... "-2.34", "2.4latest"}
default: "2.4latest"

When users override it to "0.10.0-2.24" or below, then the old metrics names / tags will still be used.

...