...
// Channel used to send events to the background thread
private BlockingQueue<RequestEvent> queue;
abstract public class RequestEvent {
private final RequestEventType eventType;
}
enum RequestEventType {
FETCH,
COMMIT,
METADATA_UPDATE,
CLOSE,
...
}
ResponseEventQueue
// Channel used to send events to the polling thread for client side execution/notification
private BlockingQueue<ResponseEvent> queue;
abstract public class ResponseEvent {
private final ResponseEventType eventType;
}
enum ResponseEventType {
ERROR,
REVOKE_PARTITIONS,
ASSIGN_PARTITIONS,
FETCH_RESPONSE,
}
...
Major Changes
Consumer Poll Changes
consumer.poll() onward will mainly be responsible for a the following tasks:
- Poll EventHandler, process the events
- Send fetches to the background thread
- Return fetched data
Major Changes
Consumer Poll Changes
Currently, the consumer.poll() does two things: poll coordinator for assignment update, auto-commit and rebalances, and fetching.
Moving forward, the consumer will only be responsible for sending and returning the fetch data and polling the channel for events such as rebalancing and error:
- Check wakeup flag. Throw WakeupException and interrupt the loop if a wakeup call was invoked.
Poll the ConsumerQueue for events
Handle exceptions
Handle callback execution
Send out fetch requests
Returning fetch results
a loop polling for events from the event handler. Here are the important events:
- Fetch Data: Return immediately
- Callback Invocation: Invoke the callback and subsequently send an acknowledgment event to the background thread to advance the state machine (see background thread section).
- Error: Handle the error upon receiving it.
ConsumerCoordinator and AbstractCoordinator
...
- Poll the EventHandler, and execute the events accordingly
- Prepare a FETCH event, and send it to the background thread.
- Wait for the CompleteableFuture to finish
- Execute the interceptor.onConsumer and return the data.
- Background thread execute the FETCH event:
- Autocommits // I think we can move the autocommit to the background thread
- issue fetch requests
...