Status

Current state: Under Discussion

Discussion thread: TBD

JIRA: KAFKA-7798

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

Today Kafka Streams embedded a few lower-level producer, consumer, and admin clients inside itself:

• A consumer client per-thread.
• A restore consumer client per-thread.
• A shared admin client per-instance.
• A producer client per-thread if EOS is turn off; otherwise a producer client per-task.

This KIP proposes to expose these embedded client's ids via the ThreadMetadata. Those clientIds are useful in a number of ways:

• KafkaStreams#metrics() includes all the metrics from its embedded clients, and are organized by MetricName's group (producer, consumer, admin) and tags (clientIds); knowing the clientIds helps to quickly find the corresponding metric from the map.
• When some of the threads have failed due to unexpected error, their embedded clients may also shutdown and be notifying the users; knowing the ids helps with such trouble shooting scenarios.
• Correlated to KIP-345, exposing the consumer client's id is useful for managing static consumer members in operations like scale-in.
• etc.

Public Interfaces

public class ThreadMetadata {

    public String consumerClientId() {
        return mainConsumerClientId;
    }

    public String restoreConsumerClientId() {
        return restoreConsumerClientId;
    }

    // NOTE: without EOS it should be a singleton; otherwise it is one clientId per owned active task
    public Set<String> producerClientIds() {
        return producerClientIds;
    }

    public String adminClientId() {
        return adminClientId;
    }

    // ... other APIs
}

Proposed Changes

As above.

Compatibility, Deprecation, and Migration Plan

• This KIP only adds a few util functions into ThreadMetadata, and hence has no impact on compatibility.

Rejected Alternatives

• None.