THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
All of the following will be added via Kafka's metrics library like most of the metrics in the Kafka brokers and other components.
Connector Metrics
Number of records (meter)
Rebalance Metrics
- rebalance duration (max, avg)
- last rebalance time
- rebalance count
kafka.connect:type=connect-coordinator-metrics,name=assigned-connectors
Briefly list any new interfaces that will be introduced as part of this proposal or any existing interfaces that will be removed or changed. The purpose of this section is to concisely call out the public contract that will come along with this feature.
A public interface is any change to the following:
Binary log format
The network protocol and api behavior
Any class in the public packages under clientsConfiguration, especially client configuration
org/apache/kafka/common/serialization
org/apache/kafka/common
org/apache/kafka/common/errors
org/apache/kafka/clients/producer
org/apache/kafka/clients/consumer (eventually, once stable)
Monitoring
Command line tools and arguments
- Anything else that will likely break existing users in some way when they upgrade
Proposed Changes
Describe the new thing you want to do in appropriate detail. This may be fairly extensive and have large subsections of its own. Or it may be a few sentences. Use judgement based on the scope of the change.
Compatibility, Deprecation, and Migration Plan
- What impact (if any) will there be on existing users?
- If we are changing behavior how will we phase out the older behavior?
- If we need special migration tools, describe them here.
- When will we remove the existing behavior?
Rejected Alternatives
Source Task Metrics
| Metric Name | Description | MBean attribute |
|---|---|---|
| source record rate | The number of records produced per second by this task belonging to the named source connector | kafka.connect:type=source-task-metrics,name=source-record-rate,connector=([-.\w]+),task=([\d]+) |
| source record total | The total number of records produced by this task belonging to the named source connector | kafka.connect:type=source-task-metrics,name=source-record-total,connector=([-.\w]+),task=([\d]+) |
| poll time percentage | The average percentage of time spent polling this task belonging to the named source connector | kafka.connect:type=source-task-metrics,name=poll-time-percentage,connector=([-.\w]+),task=([\d]+) |
| transform time percentage | The average percentage of time spent transforming source records for this task belonging to the named source connector | kafka.connect:type=source-task-metrics,name=transform-time-percentage,connector=([-.\w]+),task=([\d]+) |
| write time percentage | The average percentage of time spent converting and writing source records for this task belonging to the named source connector | kafka.connect:type=source-task-metrics,name=write-time-percentage,connector=([-.\w]+),task=([\d]+) |
| pause time percentage | The average percentage of time this task belonging to the named source connector were paused | kafka.connect:type=source-task-metrics,name=pause-time-percentage,connector=([-.\w]+),task=([\d]+) |
Source Connector Metrics
| Metric Name | Description | MBean attribute |
|---|---|---|
| source record rate | The number of records produced per second by all tasks belonging to the named source connector | kafka.connect:type=source-connector-metrics,name=source-record-rate,connector=([-.\w]+) |
| source record total | The total number of records produced by all tasks belonging to the named source connector | kafka.connect:type=source-connector-metrics,name=source-record-total,connector=([-.\w]+) |
| poll time percentage | The average percentage of time spent polling all tasks belonging to the named source connector | kafka.connect:type=source-connector-metrics,name=poll-time-percentage,connector=([-.\w]+) |
| transform time percentage | The average percentage of time spent transforming source records for all tasks belonging to the named source connector | kafka.connect:type=source-connector-metrics,name=transform-time-percentage,connector=([-.\w]+) |
| write time percentage | The average percentage of time spent converting and writing source records for all tasks belonging to the named source connector | kafka.connect:type=source-connector-metrics,name=write-time-percentage,connector=([-.\w]+) |
| pause time percentage | The average percentage of time all tasks belonging to the named source connector were paused | kafka.connect:type=source-connector-metrics,name=pause-time-percentage,connector=([-.\w]+) |
| status | The current status of the connector, one of: running, paused, stopped | kafka.connect:type=source-connector-metrics,name=status,connector=([-.\w]+) |
Sink Task Metrics
| Metric Name | Description | MBean attribute |
|---|---|---|
| sink record rate | The number of records produced per second by this task belonging to the named source connector | kafka.connect:type=sink-task-metrics,name=sink-record-rate,connector=([-.\w]+),task=([\d]+) |
| sink record total | The total number of records produced by this task belonging to the named source connector | kafka.connect:type=sink-task-metrics,name=sink-record-total,connector=([-.\w]+),task=([\d]+) |
| read time percentage | The average percentage of time spent polling this task belonging to the named source connector | kafka.connect:type=sink-task-metrics,name=read-time-percentage,connector=([-.\w]+),task=([\d]+) |
| transform time percentage | The average percentage of time spent transforming sink records for this task belonging to the named sink connector | kafka.connect:type=sink-task-metrics,name=transform-time-percentage,connector=([-.\w]+),task=([\d]+) |
| put time percentage | The average percentage of time this task belonging to the named sink connector spent putting/processing sink records | kafka.connect:type=sink-task-metrics,name=put-time-percentage,connector=([-.\w]+),task=([\d]+) |
| flush time percentage | The average percentage of time this task belonging to the named sink connector spent flushing sink records | kafka.connect:type=sink-task-metrics,name=flush-time-percentage,connector=([-.\w]+),task=([\d]+) |
| pause time percentage | The average percentage of time this task belonging to the named sink connector were paused | kafka.connect:type=sink-task-metrics,name=pause-time-percentage,connector=([-.\w]+),task=([\d]+) |
Sink Connector Metrics
| Metric Name | Description | MBean attribute |
|---|---|---|
| sink record rate | The number of records produced per second by all tasks belonging to the named source connector | kafka.connect:type=sink-connector-metrics,name=sink-record-rate,connector=([-.\w]+) |
| sink record total | The total number of records produced by all tasks belonging to the named source connector | kafka.connect:type=sink-connector-metrics,name=sink-record-total,connector=([-.\w]+) |
| read time percentage | The average percentage of time spent polling all tasks belonging to the named source connector | kafka.connect:type=sink-connector-metrics,name=read-time-percentage,connector=([-.\w]+) |
| transform time percentage | The average percentage of time spent transforming sink records for all tasks belonging to the named sink connector | kafka.connect:type=sink-connector-metrics,name=transform-time-percentage,connector=([-.\w]+) |
| put time percentage | The average percentage of time all tasks belonging to the named sink connector spent putting/processing sink records | kafka.connect:type=sink-connector-metrics,name=put-time-percentage,connector=([-.\w]+) |
| flush time percentage | The average percentage of time all tasks belonging to the named sink connector spent flushing sink records | kafka.connect:type=sink-connector-metrics,name=flush-time-percentage,connector=([-.\w]+) |
| pause time percentage | The average percentage of time all tasks belonging to the named sink connector were paused | kafka.connect:type=sink-connector-metrics,name=pause-time-percentage,connector=([-.\w]+) |
| status | The current status of the connector, one of: running, paused, stopped | kafka.connect:type=sink-connector-metrics,name=status,connector=([-.\w]+) |
| partition count | The number of topic partitions assigned to all tasks running in this worker for the named sink connector | kafka.connect:type=sink-connector-metrics,name=partition-count,connector=([-.\w]+) |
Worker Metrics
| Metric Name | Description | MBean attribute |
|---|---|---|
assigned tasks | The number of tasks run in this worker (existing metric) | kafka.connect:type=connect-coordinator-metrics,name=assigned-tasks,worker=([-.\w]+) |
| assigned connectors | The number of connectors run in this worker (existing metric) | kafka.connect:type=connect-coordinator-metrics,name=assigned-connectors,worker=([-.\w]+) |
task count | The number of tasks run in this worker | kafka.connect:type=connect-worker-metrics,name=assigned-tasks,worker=([-.\w]+) |
| connector count | The number of connectors run in this worker | kafka.connect:type=connect-worker-metrics,name=assigned-connectors,worker=([-.\w]+) |
| sink record rate | The number of sink records consumed per second by all sink connectors | kafka.connect:type=connect-worker-metrics,name=sink-record-rate,worker=([-.\w]+) |
| sink record total | The total number of sink records consumed by all sink connector | kafka.connect:type=connect-worker-metrics,name=sink-record-total,worker=([-.\w]+) |
| source record rate | The number of source records produced per second by all source connectors | kafka.connect:type=connect-worker-metrics,name=source-record-rate,worker=([-.\w]+) |
| source record total | The total number of source records produced by all source connector | kafka.connect:type=connect-worker-metrics,name=source-record-total,worker=([-.\w]+) |
| leader name | The name of the group leader | kafka.connect:type=connect-worker-metrics,name=leader-name,worker=([-.\w]+) |
| state | The state of this worker, one of: rebalancing, running | kafka.connect:type=connect-worker-metrics,name=state,worker=([-.\w]+) |
| offset commit success total | The total number of successful offset commits | kafka.connect:type=connect-worker-metrics,name=offset-commit-success-total,worker=([-.\w]+) |
| offset commit success percentage | The average percentage of offset commits that succeeded | kafka.connect:type=connect-worker-metrics,name=offset-commit-success-percentage,worker=([-.\w]+) |
| offset commit failure total | The total number of failed offset commits | kafka.connect:type=connect-worker-metrics,name=offset-commit-failure-total,worker=([-.\w]+) |
| offset commit failure percentage | The average percentage of offset commits that failed | kafka.connect:type=connect-worker-metrics,name=offset-commit-failure-percentage,worker=([-.\w]+) |
| offset commit total | The total number of offset commits | kafka.connect:type=connect-worker-metrics,name=offset-commit-total,worker=([-.\w]+) |
| offset commit maximum time | The maximum time spent to commit offsets | kafka.connect:type=connect-worker-metrics,name=offset-commit-max-time,worker=([-.\w]+) |
| offset commit 99th percentile time | The 99th percentile time spent to commit offsets during the last window (defaults to an hour) | kafka.connect:type=connect-worker-metrics,name=offset-commit-99p-time,worker=([-.\w]+) |
| offset commit 95th percentile time | The 95th percentile time spent to commit offsets during the last window (defaults to an hour) | kafka.connect:type=connect-worker-metrics,name=offset-commit-95p-time,worker=([-.\w]+) |
| offset commit 90th percentile time | The 90th percentile time spent to commit offsets during the last window (defaults to an hour) | kafka.connect:type=connect-worker-metrics,name=offset-commit-90p-time,worker=([-.\w]+) |
| offset commit 75th percentile time | The 75th percentile time spent to commit offsets during the last window (defaults to an hour) | kafka.connect:type=connect-worker-metrics,name=offset-commit-75p-time,worker=([-.\w]+) |
Worker Rebalance Metrics
| Metric Name | Description | MBean attribute |
|---|---|---|
| rebalance success total | The total number of successful rebalances | kafka.connect:type=connect-worker-rebalance-metrics,name=rebalance-success-total,worker=([-.\w]+) |
| rebalance success percentage | The average percentage of rebalances that succeeded | kafka.connect:type=connect-worker-rebalance-metrics,name=rebalance-success-percentage,worker=([-.\w]+) |
| rebalance failure total | The total number of failed rebalances | kafka.connect:type=connect-worker-rebalance-metrics,name=rebalance-failure-total,worker=([-.\w]+) |
| rebalance failure percentage | The average percentage of rebalances that failed | kafka.connect:type=connect-worker-rebalance-metrics,name=rebalance-failure-percentage,worker=([-.\w]+) |
| rebalance maximum time | The maximum time spent to rebalance | kafka.connect:type=connect-worker-rebalance-metrics,name=rebalance-max-time,worker=([-.\w]+) |
| rebalance 99th percentile time | The 99th percentile time spent to rebalance during the last window (defaults to an hour) | kafka.connect:type=connect-worker-rebalance-metrics,name=rebalance-99p-time,worker=([-.\w]+) |
| rebalance 95th percentile time | The 95th percentile time spent to rebalance during the last window (defaults to an hour) | kafka.connect:type=connect-worker-rebalance-metrics,name=rebalance-95p-time,worker=([-.\w]+) |
| rebalance 90th percentile time | The 90th percentile time spent to rebalance during the last window (defaults to an hour) | kafka.connect:type=connect-worker-rebalance-metrics,name=rebalance-90p-time,worker=([-.\w]+) |
| rebalance 75th percentile time | The 75th percentile time spent to rebalance during the last window (defaults to an hour) | kafka.connect:type=connect-worker-rebalance-metrics,name=rebalance-75p-time,worker=([-.\w]+) |
| time since last rebalance | The time since the most recent rebalance | kafka.connect:type=connect-worker-rebalance-metrics,name=time-since-last-rebalance,worker=([-.\w]+) |
| task failure rate | The number of tasks that failed in this worker | kafka.connect:type=connect-worker-rebalance-metrics,name=task-failure-rate,worker=([-.\w]+ |
Worker REST Metrics
| Metric Name | Description | MBean attribute |
|---|---|---|
| REST request rate | The number of requests handled by the REST endpoints | kafka.connect:type=worker-rest-metrics,name=request-rate,worker=([-.\w]+) |
kafka.connect:type=connect-coordinator-metrics,name=assigned-connectors
Proposed Changes
We will add the relevant metrics as specified in the Public Interfaces section.
Compatibility, Deprecation, and Migration Plan
We are introducing new metrics so there is no compatibility impact. Note that two existing metrics exist but will not be changed.
Rejected Alternatives
The following metrics were considered but were rejected:If there are alternative ways of accomplishing the same thing, what were they? The purpose of this section is to motivate why the design is the way it is and not some other way.