...

To coordinate the polling thread and background thread  threads to execute the callbacks and acknowledge the completion, we need to modify the existing coordinator states.. In particular, PREPARING_REBALANCE and ASSIGN_PARTITION. It is because of the asynchronous nature of this threading model, i.e., the background thread needs to wait for an acknowledgment event from the callback invocation to proceed with the rebalancing safely.

1. PREPARING_REBALANCE  → We split the PREPARING_REBALANCE into PREPARE_REVOCATION, and PARTITION_REVOKED, PREPARING_REBALANCE

   1. PREPARE_REVOCATION: Pause partitions that will be revoked.

   2. REVOKING_PARTITION: Await for the onPartitionRevoked to be completed.

   3. PARTITION_REVOKED: Update the subscription.  Autocommit.

2. And ASSIGN_PARTITION state will be added before STABLE

   1. ASSIGNING_PARTITIONS: Await for onPartitionAssign to be completed.

   2. Upon onPartitionAssign's completion, we will move the state to STABLE.

Points of consideration:

• What is revocation timed out? 
  • Retry, then leave group 
  • Directly leave group
  • Or continuously retry?

After the proposed state transition is as such:

...

1. Send a leave group event
2. unsubscribe from the topics

Major Changes

Fetcher

We will break the current fetcher into three separate parts to accommodate the asynchronous design, i.e., we need the background thread to send fetches autonomously and the polling thread to collect fetches when these fetches become available. We will have 3 separate classes here:

1. FetchSender: Responsible for sending fetches in the background thread
2. FetchHandler: Sitting in the polling thread's poll loop, processing the fetch response from the fetch event. 
3. FetchBuffer: This is the CompletedFetches in the old implementation. The use case prevents the FetchSender from sending too many fetches and causing memory issues. This will be removed once we implement the memory-based buffer.(KIP-81)

Consumer Poll Changes

consumer.poll() will mainly be a loop polling for events from the event handler.  Here are the important events:

...

The new consumer should be backward compatible.

Alternative Proposals

Fetcher

• Should some of the fetcher configuration be dynamic

• Configurable prefetch buffer

...