Status

Current state: Under Discussion

Discussion thread: here [Todo]

JIRA: here

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

Connect framework provides good metrics for monitoring behavior. Currently, we have a metric on the Worker to measure number of tasks: .

It is useful in many applications to measure the number of tasks on a Connector. For example, an administrator may wonder the average number of tasks per connector, which types of connectors create the most tasks, if some connectors are outliers (creating too many or too few tasks). The current metric providing tasks per Worker is useful in its own right, but does not address any of these needs.

Public Interfaces

We propose adding the new metric "connector-task-count" on the group name "connector-metrics" in the ConnectMetricsRegistry.

Proposed Changes

A new Connector level metric will be added, "connector-task-count". This metric will be calculated from the `Herder::connectorInfo` method. In order to enable this change, the `WorkerConnector` will be constructed with a `Herder` to have access to the task statuses.

A PR is available here: 

Compatibility, Deprecation, and Migration Plan

This KIP simply adds a new metric.

Rejected Alternatives

Naming it simply "task-count"

"connector-task-count" keeps consistent with the existing names "connector-version", "connector-type", and "connector-status". Further, while not strictly necessary, it does disambiguate from the Worker's "task-count" metric name.

Rely on the REST API

While the Connect REST API does provide the number of tasks for a connector, it is unuseful in Standalone Mode. Further, exposing this number through the common metrics interface enables downstream interfaces and pipelines to gather metrics from a single source.