Versions Compared

Old Version 23

changes.mady.by.user guozhang Wang

Saved on

compared with

New Version Current

changes.mady.by.user Bruno Cadonna

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

JIRA:

Jira
serverASF JIRA
columnskey,summary,type,created,updated,due,assignee,reporter,priority,status,resolution
maximumIssues20
jqlQuerykey = KAFKA-6820 or key = KAFKA-6819 or key = KAFKA-7963 or key = KAFKA-5676 8856 or key = KAFKA-9230 or key = KAFKA-8856 9753
serverId5aa69414-a9e9-3523-82ec-879b028fb15b

...


LEVEL 0LEVEL 1LEVEL 2LEVEL 3LEVEL 3LEVEL 3

Per-Client

Per-Thread

Per-Task 

Per-Processor-Node Per-State-StorePer-Cache
TAGS

type=stream-metrics,client-id=[client-id]

type=stream-thread-metrics,thread-id=[threadId]


(! tag name changed)

type=stream-task-metrics,thread-id=[threadId],task-id=[taskId]


(! tag name changed)

type=stream-processor-node-metrics,thread-id=[threadId],task-id=[taskId],processor-node-id=[processorNodeId]


(! tag name changed)

stream-state-metrics,thread-id=[threadId],task-id=[taskId],[storeType]-state-id=[storeName]


(! tag name changed)

type=stream-record-cache-metrics,thread-id=[threadId],task-id=[taskId],record-cache-id=[storeName]


(! tag name changed)

version | commit-id (static gauge)
INFO ($)




application-id (static gauge)
INFO ($)




topology-description (static gauge)
INFO ($)




state (dynamic gauge)
INFO ($)




alive-stream-threads (dynamic gauge)
INFO ($)




process-latency (avg | max)

INFODEBUG(! removed for now)

process (rate | total)

INFODEBUG ( → ) on source-nodes onlyDEBUGDEBUG on source-nodes only

punctuate-latency (avg | max)

INFODEBUG


punctuate (rate | total)

INFODEBUG


commit-latency (avg | max)

INFODEBUG


commit (rate | total)

INFODEBUG


poll-latency (avg | max)

INFO



poll (rate | total)

INFO



process | punctuate | commit | poll-ratio (dynamic gauge)

INFO



task-created | closed (rate | total)

INFO



activepoll-records (avg | max)

INFO



process-records (avg | max)

INFO



active-process-ratio (process-ratio (dynamic gauge)


INFO ($) (percentage of time the hosting thread is spending with this active task)


standby-process-ratio (dynamic gauge)


INFO ($) (percentage of time the hosting thread is spending with this standby task)


dropped-records (rate | total)


INFO ($) (number of records dropped within this task due to all kinds of scenarios)




active-buffer-count (dynamic gauge)


DEBUG


enforced-processing (rate | total)


DEBUG


record-lateness (avg | max)


DEBUG


suppression-emit (rate | total)



DEBUG * (suppress processor only)

suppression-buffer-size (avg | max)




DEBUG * (suppression buffer only)
suppression-buffer-count (avg | max)




DEBUG * (suppression buffer only)
(put | put-if-absent .. | get)-latency (avg | max)




DEBUG * (excluding suppression buffer)

                 (! name changed)


(put | put-if-absent .. | get) (rate)




DEBUG * (excluding suppression buffer)

                 (! name changed)


hit-ratio (avg | min | max)





DEBUG  (! name changed)

...

Code Block
late-records-drop: INFO at processor node level, replaced by INFO task-level "dropped-records".

skipped-records: INFO at thread and processor node level, replaced by INFO task-level "dropped-records".

expired-window-record-drop: DEBUG at state store level, replaced by INFO task-level "dropped-records".

forward-rate: DEBUG at processor-node level, replaced by DEBUG processor node level "process-rate".

destorydestroy-rate: DEBUG at processor-node level, covered by INFO thread-level "task-closed-rate".

create-rate: DEBUG at processor-node level, covered by INFO thread-level "task-create-rate".


Release History

2.4

  • client-level metrics:
    • added:
      • version 
      • commit-id
      • application-id
      • topology-description
      • state

2.5

  • thread-level metrics:
    • refactored:
      • process-latency (avg | max)
      • process (rate | total) 
      • punctuate-latency (avg | max)
      • punctuate (rate | total)
      • commit-latency (avg | max)
      • commit (rate | total)
      • poll-latency (avg | max)
      • poll (rate | total)
      • task-created | closed (rate | total)
    • removed:
      • skipped-records
  • task-level metrics:
    • refactored:
      • process-latency (avg | max)
      • process (rate | total) 
      • punctuate-latency (avg | max)
      • punctuate (rate | total)
      • commit-latency (avg | max)
      • commit (rate | total)
      • enforced-processing (rate | total)
      • record-lateness (avg | max)
    • added:
      • dropped-records (rate | total)
    • removed:
      • expired-window-record-drop
  • processor-node-level:
    • refactored:
      • process (rate | total)
      • suppression-emit (rate | total)
    • removed:
      • process-latency (avg | max)
      • late-records-drop
      • skipped-records
      • forward-rate
      • destroy-rate
      • create-rate
  • state-store-level:
    • refactored:
      • suppression-buffer-size (avg | max)
      • suppression-buffer-count (avg | max)
      • (put | put-if-absent .. | get)-latency (avg | max)
      • (put | put-if-absent .. | get) (rate)
    • removed:
      • expired-window-record-drop
  • cache-level:
    • refactored:
      • hit-ratio (avg | min | max)

2.6

  • client-level metrics:
    • added: 
      • alive-stream-threads
  • thread-level metrics:
    • added: 
      • process | punctuate | commit | poll-ratio
      • poll-records (avg | max)
      • process-records (avg | max)
  • task-level metrics:
    • added: 
      • active-process-ratio

Proposed Changes

As above.

...