...

If you are a Kafka user and you want to track the progress a client is making you can use the returned committed offset. However Streams does not have that capability as several Kafka clients are embedded in Streams client. This KIP proposes to add methods to Kafka Streams TaskMetadata that report the progress Kafka exposes already.

...

package org.apache.kafka.streams.processor;
  
KafkaStreamsTaskMetadata.java
/**
* Returns the last committed offset for each TopicPartition in a task assigned to this client.
*/
public Map<TopicPartition, Long> committedOffsets();

/**
* Returns a map telling if each task is idling.
*
*/
public Map<TaskId, boolean> tasksIdling()

Proposed Changes

committedOffsets

This should have the latest committed off set of each task assigned to the client. Each poll will update this. The offsets committed by the task manger will be saved. Each time a new commit is made the map containing the offsets will update. When committedOffsets() is called then it will return this map.

isTaskIdling

public Map<TopicPartition, Long> endOffsets();
public Long timeCurrentIdlingStarted();

Proposed Changes

In order to make these more accessible we will allow localThreadMetadata to be called in all states, not just running or rebalancing.

committedOffsets

This method will return a map that contains a mapping of all TopicPartitions and their committed offsets. The committed offset will be the highest seen so far.

endOffsets

Similarly this method will return a map that contains a mapping of all TopicPartitions and their end offsets.

timeCurrentIdlingStarted

This will record the time the task started idling. Once the task has stopped idling it will reset to -1. This also lets the user verify if the task is idling properly as they can compare the time with the config for the max idling time.If a task assigned to this instance is intentionally not pulling because it is idling this method will report that

Compatibility, Deprecation, and Migration Plan

...

• reporting these as metrics
• adding a committed offset method to kafka streams