Status

Current state: Under Discussion"Accepted

Discussion thread: here [Change the link from the KIP proposal email archive to your own email thread]

JIRA: 

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

...

Public Interfaces

New Metrics

Kafka Streams

blocked-time-total
tags: thread-id, application-id
group: stream-thread-metrics
level: INFO. The proposed metrics should be collectible at INFO level without adding meaningful overhead. They require sampling the time twice during the corresponding API calls, which nowadays is very cheap.
description: the total time the Kafka Streams thread spent blocked on Kafka.

thread-start-time
tags: thread-id
group: stream-thread-metrics
level: INFO
description: the epoch time the Kafka Streams thread was started. This is useful for computing the processing ratio during the first interval after the thread starts. 

Producer

flush-time-ns-total
tags: threadclient-id, application-id
group: producer-metrics
level: INFO
description: the total time the Kafka Streams thread Producer spent in `Producer.flush` in nanoseconds.

txn-init-time-ns-total
tags: client-id
group: producer-metrics
level: INFO
description: the total time the Producer spent initializing transactions in nanoseconds (for EOS).

txn-commitbegin-time-ns-total
tags: thread-id, application-idclient-id
group: producer-metrics
level: INFO
description: the total time the Producer spent in beginTransaction in nanoseconds (for EOS).

txn-send-offsets-time-ns-total
tags: client-id
group: producer-metrics
level: INFO
description: the total time the Kafka Streams thread spent committing transactions Producer spent sending offsets to transactions in nanoseconds (for EOS).

offsettxn-commit-time-ns-total
tags: threadclient-id, application-id
group: producer-metrics
level: INFO
description: the total time the Kafka Streams thread Producer spent committing offsets transactions in nanoseconds (for AOSEOS).

threadtxn-startabort-time-ns-total
tags: thread-id, application-id
deciption: the epoch time the Kafka Streams thread was started. This is useful for computing utilization during intervals close to the stream thread start timeclient-id
group: producer-metrics
level: INFO
description: the total time the Producer spent aborting transactions in nanoseconds (for EOS).

Consumer

commited-time-ns-total
tags: client-id
group: consumer-metrics
level: INFO
description: the total time the Consumer spent in committed in nanoseconds.

commit-sync-time-ns-total
tags: client-id
group: consumer-metrics
level: INFO
description: the total time the Consumer spent committing offsets in nanoseconds (for AOS).

Proposed Changes

flush-time-ns-total: this will be a Producer metric computed as the cumulative sum of time elapsed during calls to Producer.flush.

txn-init-time-ns-total: this will be a Producer metric computed as the cumulative sum of time elapsed during calls to Producer.initTransactions.

txn-begin-time-ns-total: this will be a Producer metric computed as the cumulative sum of time elapsed during calls to Producer.beginTransaction.

txn-send-offsets-time-ns-total: this will be a Producer metric computed as the cumulative sum of time elapsed during calls to Producer.sendOffsetsToTransaction.

txn-commit-time-ns-total: this will be a Producer metric computed as the cumulative sum of time elapsed during calls to StreamsProducer.commitTransaction from TaskManager. Producer.commitTransaction.

txn-abort-time-ns-total: this will be a Producer metric computed as the cumulative sum of time elapsed during calls to Producer.abortTransaction.

commited-time-ns-total: this will be a Consumer metric computed as the cumulative sum of time elapsed during calls to Consumer.committed.

commit-sync-time-nsoffset-commit-time-total: this will be a Consumer metric computed as the cumulative sum of time elapsed during calls to Consumer.commitSync from TaskManager.

blocked-time-ns-total: this will be a Value that returns the sum of the following metrics:

• consumer’s io-waittime-total
• consumer’s iotime-total
• consumer’s committed-time-ns-total
• consumer’s commit-sync-time-ns-total
• restore consumer’s io-waittime-total
• restore consumer’s iotime-total
• admin client’s io-waittime-total
• admin client’s iotime-total
• producer’s bufferpool-wait-time-total
• producer's flush-time-ns-total
• producer's txn-init-time-ns-total
• producer's txn-begin-time-ns-total
• producer's txn-send-offsets-time-ns-total
• producer's txn-commit-time-ns-total
• producer's txn-

...

• abort-time-ns-total

Compatibility, Deprecation, and Migration Plan

...