Page History

...

Current state: Under Discussion

Discussion thread: here [Change the link from the KIP proposal email archive to your own email thread]

JIRA: No jira.

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

...

Currently, there are 2 approaches to use the Kafka java client correctly, that is, in a way that does not cause record duplication during normal operation. (Duplicates are still possible due to failures.)

Approach 1 - process all records immediately (synchronous)

This approach is useful when all processing can happen between the calls to poll.

...

1. a poll loop which:
   1. calls consumer.poll
   2. process the received records
   3. commits offsets for the just processed records
   4. repeat

The big advantage of this approach is that it is the simplest. This is also the only approach that can be used with auto-commit.

However, this approach has the downside that asynchronous processing can not be used. Approach 2 does allow thisTherefore, we do not consider it further in this KIP.

Approach 2 - asynchronous processing

...

In the new approach, when a revoke is requested in one poll, the revoke will be completed in the next poll.

When the call to a poll invocation completes and a partition revoke is requested, the user has a choice:

...