Page History

...

• metrics  - a comma-separated list of metric name prefixes, e.g., "client.producer.partition., client.io.wait". Whitespaces are ignored.
• interval  - metrics push interval in milliseconds. Defaults to 5 minutes if not specified.
• match_<selector>  - Client matching selector that is evaluated as an anchored regexp (i.e., "something.*" is treated as "^something.*$"). Any client that matches all of the match_..  selectors will be eligible for this metrics subscription. Initially supported selectors:
  • client_instance_id - CLIENT_INSTANCE_ID UUID string representation.
  • client_id  - client's reported client.id in the GetTelemetrySubscriptionsRequest.
  • client_software_name  - client software implementation name.
  • client_software_version  - client software implementation version.
  • client_source_address  - client connection's source address from the broker's point of view.
  • client_source_port  - client connection's source port from the broker's point of view.

...

The Kafka operator adds a metrics subscription for metrics matching prefix “org.apache.kafka.client.consumer.” and with the client_source_address and client_id as metrics matching selector. Since this is a live troubleshooting case the metrics push interval is set to a low 10 seconds.

The metrics subscription propagates through configuration change notifications to all brokers  brokers which update their local metrics subscription config cache and regenerates the SubscriptionId.

Upon the next intervalled PushTelemetryRequest or GetTelemetrySubscriptionsRequest from the consumer the clients matching the given source address and client_id the receiving broker sees that the SubscriptionId no longer matches its metrics subscription cache, the client retrieves the new metrics subscription and schedules its next metrics push to a random value between PushIntervalMs * 0.5 .. PushIntervalMs * 1.5.

...

Multiple consumers from the same source address and using the same client_id may now be pushing metrics to the cluster. The support system starts receiving the metrics and soon finds a metric push from the desired client.id which now provides a mapping from client.id to client_instance_id. At this point the metrics subscription may be altered to only match the displays the matching client metrics to the operator. If the operator is able to further narrow down the client instance through other information in the metrics it may alter the metrics subscription to match on that client's client_instance_id of the matching client. But in either case the metrics matching the given client.id are displayed to the operator.

...