Status

Current state: Under Discussion

Discussion thread:  here

JIRA: here

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

Eversince KIP-266 was concluded, several new versions of KafkaConsumer's methods has been introduced. Meanwhile, older versions has been deprecated in favor of these newer methods which allows us to control how much time each individual method blocks. Therefore, there has been a pressing need to migrate Kafka Streams from these old API to newer ones. It has also been noted that some of the older methods which Kafka Streams call blocks infinitely, migrating Kafka Streams will help solve that issue.

Public Interfaces

Currently, there are a couple of options that are available to us:

1. A couple of old configurations were related with blocking in Streams: RETRIES_CONFIG and RETRY_BACKOFF_MS_CONFIG. However, it has been noted that the application of these two configs has been relatively restricted, mainly being used in global consumer's methods (per current design).  We might remove these configs in favor of a larger timeout.
2. Another concern is the introduction of another config for blocking. We might use a global configuration (similar to request.timeout.ms) or allow a user-specified timeout which is passed in via methods (the latter option was favored over the former in KIP-266).

Proposed Changes

We will change the KafkaConsumer methods used in Kafka Streams to the newer methods created in KIP-266.

Compatibility, Deprecation, and Migration Plan

The older methods will be deprecated in favor of the newer ones found in Kafka Streams if  the user-specified timeout approach is found to be more applicable to this KIP.

Rejected Alternatives

If there are alternative ways of accomplishing the same thing, what were they? The purpose of this section is to motivate why the design is the way it is and not some other way.