Status

Current state: Adopted (2.2.0)

Discussion thread: TBD

JIRA: KAFKA-7798

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

Today Kafka Streams embedded a few lower-level producer, consumer, and admin clients inside itself:

  • A consumer client per-thread.
  • A restore consumer client per-thread.
  • A shared admin client per-instance.
  • A producer client per-thread if EOS is turn off; otherwise a producer client per-task.

This KIP proposes to expose these embedded client's ids via the ThreadMetadata. Those clientIds are useful in a number of ways:

  • KafkaStreams#metrics() includes all the metrics from its embedded clients, and are organized by MetricName's group (producer, consumer, admin) and tags (clientIds); knowing the clientIds helps to quickly find the corresponding metric from the map.
  • When some of the threads have failed due to unexpected error, their embedded clients may also shutdown and be notifying the users; knowing the ids helps with such trouble shooting scenarios.
  • Correlated to KIP-345, exposing the consumer client's id is useful for managing static consumer members in operations like scale-in.
  • etc.

Public Interfaces

public class ThreadMetadata {

    public String consumerClientId() {
        return mainConsumerClientId;
    }

    public String restoreConsumerClientId() {
        return restoreConsumerClientId;
    }

    // NOTE: without EOS it should be a singleton; otherwise it is one clientId per owned active task
    public Set<String> producerClientIds() {
        return producerClientIds;
    }

    public String adminClientId() {
        return adminClientId;
    }

    // ... other APIs
}



Proposed Changes

As above.

Compatibility, Deprecation, and Migration Plan

  • This KIP only adds a few util functions into ThreadMetadata, and hence has no impact on compatibility.

Rejected Alternatives

  • None.
  • No labels