...

This proposal aims to provide a generic and standardised interface through which a cluster operator or an external system may request and a client can push client-side metrics to the broker, a plugin interface for how those metrics are handled by the broker, and a minimum set of standard pre-defined metrics that a Kafka client should expose through this interface. A supporting client should have metrics enabled by default and not require any extra configuration, and will provide at least the standardised metrics as outlined later in this proposal.

One of the key goals of this KIP is to have the proposed metrics and telemetry interface generally available and enabled by default in all of the mainstream Kafka clients, allowing troubleshooting and monitoring as needed without interaction from cluster end-users (user privacy is an important concern and extra care is taken in this proposal).

Public Interfaces

This KIP introduces the following new public interfaces:

...

Not relevant.

Rejected Alternatives

Send metrics out-of-band directly to collector

There are plenty of existing solutions that allows a client implementation to send metrics directly to a collector, but it falls short to meet the enabled-by-default requirements of this KIP:

• will require additional client-side configuration: endpoints, authentication, etc.
• may require additional network filtering and our routing configuration to reach the endpoints. By using the Kafka protocol we already have a usable connection.
• adds another network protocol the client needs to handle and its runtime dependencies (libraries..).

Dedicated Metrics coordinator based on client instance id

...