Page History

...

1. FetchSender: Responsible for sending fetches in the background thread
2. FetchHandler: Sitting in the polling thread's poll loop, processing the fetch response from the fetch event. 
3. FetchBuffer: This is the CompletedFetches in the old implementation. The use case prevents the FetchSender from sending too many fetches and causing memory issues. This will be removed once we implement the memory-based buffer.(KIP-81)

Consumer Poll Changes

We will remove the metadata logic from the consumer.poll() will mainly be a loop polling for events from the event handler.  Here are the important events:, so that the execution of the poll loop is much simplified. It mainly handles:

1. fetches
2. callback invocation
3. errors
4. Fetch Data: Return immediately
5. Callback Invocation: Invoke the callback and subsequently send an acknowledgment event to the background thread to advance the state machine (see background thread section).
6. Error: Handle the error upon receiving it.

ConsumerCoordinator and AbstractCoordinator

• New states will be introduced (see Rebalance States section above).  The main purpose is to make the background thread drive the poll, and letting the polling thread to invoke the callbacks.

• Remove HeartbeatThread. Therefore, we won't be starting the heartbeat thread.

  • It will still require a fetch event to poll heartbeat.  As only polling thread issues fetch events, and we want to respect the existing implementation.
• Timeouts and timers will be reevaluated and possibly removed.
• while loops will be reevaluated and possibly thrown out.  In the new implementation the coordinator will be non-blocking, and its states are managed by the background thread loop.

Wakeup() and WakeupException

...

wakeup()

In the new design, wakeup will interrupt the blocking polling loop in the polling thread.

Timeout Policy

Consumer.poll() - user provide timeout

...

CommitOffsetSync: user provided

Rebalance State Timeout: maybe using the request timeout

Is there a better way to configure session interval and heartbeat interval?

...