You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 11 Next »

This page is meant as a template for writing a KIP. To create a KIP choose Tools->Copy on this page and modify with your content and replace the heading with the next KIP number and a description of your issue. Replace anything in italics with your own description.

Status

Current state: "Under Discussion"

Discussion thread: here [Change the link from the KIP proposal email archive to your own email thread]

JIRA: here and here

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

If you are a Kafka user and you want to track the progress a client is making you can use the returned committed offset. However Streams does not have that capability as several Kafka clients are embedded in Streams client. This KIP proposes to add methods to TaskMetadata that report the progress Kafka exposes already.

Public Interfaces

package org.apache.kafka.streams.processor;
TaskMetadata.java

public Map<TopicPartition, Long> committedOffsets();
public Map<TopicPartition, Long> endOffsets();
public Long timeCurrentIdlingStarted();


Proposed Changes

In order to make these more accessible we will allow localThreadMetadata to be called in all states, not just running or rebalancing.

committedOffsets

This method will return a map that contains a mapping of all TopicPartitions and their committed offsets. The committed offset will be the highest seen so far.

endOffsets

Similarly this method will return a map that contains a mapping of all TopicPartitions and their end offsets.

timeCurrentIdlingStarted

This will record the time the task started idling. Once the task has stopped idling it will reset to -1. This also lets the user verify if the task is idling properly as they can compare the time with the config for the max idling time.


Compatibility, Deprecation, and Migration Plan

Since methods are only added and no other methods are modified, this KIP should not

  • affect backward-compatibility
  • deprecate public interfaces
  • need a migration plan other than adding the new metrics to its own monitoring component

Rejected Alternatives

  • reporting these as metrics
  • adding a committed offset method to kafka streams
  • No labels