Status

Current state: "Under Discussion"

Discussion thread: here

JIRA: here

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

In KIP-266, Kafka deprecated `poll(long)` with `poll(Duration)` as former method applies no timeout on assignment metadata update. New method applies timeout value to do both "update assignment metadata" as well as "poll some records", which doesn't cover the case where caller is only interested in assignment metadata. `poll(0)` has been used as some kind of hack for such purpose (though Kafka document clarifies the behavior is not guaranteed), and there's no alternative - `poll(Duration.ZERO)` would bring timeout on updating assignment metadata.

This KIP will propose a new public method to only trigger updating assignment metadata.

Public Interfaces

This KIP adds below method to KafkaConsumer:

boolean updateAssignmentMetadata(Duration timeout);

`updateAssignmentMetadata` will be served as a new method to only update an assignment, fetches offsets, etc instead of pulling records. This method will return true if the method succeeds to update within timeout, and return false if not.

Proposed Changes

The real intention of this KIP is exposing `KafkaConsumer.updateAssignmentMetadataIfNeeded` to the public API, with renaming to reduce the length of method name a bit. Name suffix like `IfNeeded` is appropriate to private method, so good to remove this suffix if we expose the method. Many places of Kafka test codes already leverage `KafkaConsumer.updateAssignmentMetadataIfNeeded` for similar purpose, and this KIP are proposing to expose this as public API. 

Compatibility, Deprecation, and Migration Plan

There's no impact on existing users as we only add new method. End users who leverages `poll(0)` are encouraged to migrate to the new method with proper timeout (and retry), as it provides expected behavior with avoiding infinite hang.

Rejected Alternatives

Add overloaded version of `poll(Duration, Duration)` to provide timeout for both updating assignment metadata and pulling records

While this method helps to achieve similar effect on `poll(0)`, this still doesn't guarantee the desired behavior - no records will be polled. As end users only interest with assignment metadata when they call `pull(0)`, it would be better to provide a method which guarantee such behavior.