...

• Polling thread: handles all of the API requests and callback executions.

• Background thread: handles network communication such as heartbeat, coordinator requests, and rebalance flow execution.

• Communication channels: which are responsible for:

  • ServerEventQueueRequestEventQueue: sending events to the background thread

  • ConsumerEventQueueResponseEventQueue: sending events to the polling thread

...

• The object is frequently accessed by both the polling thread and background thread.
• The object often requires instaneous response; otherwise, the performance will be heavily impacted
• Also, please review the rejected proposals section below. I talked about a few ideas that we've had.

Image RemovedImage Added

Important Components

...

We use a blocking queue to send API events from the polling thread to the background thread.  The queues will be abstract into an eventHandler, so we don't need to operate directly with the queue.

...

EventHandler

interface EventHandler {
  public ResponseEvent poll();
  public void add(RequestEvent event);
}

RequestEventQueue

// Channel used to send events to the background thread

private BlockingQueue<ServerEvent>BlockingQueue<RequestEvent> queue;

abstract public class ServerEventRequestEvent {
   private final ServerEventTypeRequestEventType eventType;
}

enum ServerEventTypeRequestEventType {
   FETCH,
   COMMIT,
   METADATA_UPDATE,
   CLOSE,
   ...
}

...

ResponseEventQueue

// Channel used to send events to the polling thread for client side execution/notification

private BlockingQueue<ConsumerEvent>BlockingQueue<ResponseEvent> queue;

abstract public class ConsumerEventResponseEvent {
   private final ConsumerEventTypeResponseEventType eventType;
}

enum ConsumerEventTypeResponseEventType {
   ERROR,
   REVOKE_PARTITIONS,
   ASSIGN_PARTITIONS,
}

...

consumer.poll() onward will be responsible for a the following tasks:

1. Poll ConsumerEventQueueEventHandler, execute the events
2. Build FetchEvent, and send it to the background thread
3. Return fetched data

...

1. The polling thread updates the subscriptionState.
2. Create a SUBSCRIBE event, and send it to the background thread.
3. Background thread executes the event:
   1. Update metadata

Poll

1. Poll the ConsumerEventQueueEventHandler, and execute the events accordingly
2. Prepare a FETCH event, and send it to the background thread.
3. Wait for the CompleteableFuture to finish
4. Execute the interceptor.onConsumer and return the data.
5. Background thread execute the FETCH event:
   
   1. Autocommits // I think we can move the autocommit to the background thread
   2. issue fetch requests

...