...

• Define the responsibilities of the polling thread and the background thread
• Define the communication interface and protocol between the two threads
• Define a Network layer over the NetworkClient, including how different requests are sent and handled
  • Define RequestManagers
  • Define RequestStates to handle retry backoff
• Redesign SubscriptionState
• Deprecate HeartbeatThread and move it to the background thread
• Define rebalance flow, including how JoinGroup and SyncGroup request is sent and how callbacks are triggered

• Refactor the KafkaConsumer API innards

• Address issues in these Jira tickets

Terms

Polling Thread

The client thread, or the polling thread, is where the thread user invokes the poll() operation.

Background Thread

Specifically, we use this generic term in the new design context to indicate the active process running in the background. It handles network communication, rebalancing, and heartbeat.

Heartbeat Thread

The background running thread in the current implementation (see KIP-62).

Channel

The communication interface between two threads. One channel delivers messages from the polling thread to the background thread. The other channel delivers messages submitted by the background thread to the polling thread.

Event

The primary communication medium for the polling thread is to interact with the background thread.

Design

The new consumer is asynchronous and event drive. Former means we are separating the responsibility between the polling thread and the background thread. The latter means the two threads communicate via events. The core components are:

...

If the Consumer API is a blocking call, the event passed from the application thread to the network thread will include an embedded CompleteableFuture. After enqueuing the event, the application thread will invoke Future.get(), effectively blocking itself until a result is provided. When the result for the Consumer API call is ready, the network thread will then invoke CompeteableFuture.complete() with the result, allowing the application thread to continue execution.

Components of the Network Thread

Terminologies:

• CB: Callbacks registered and invoked on the polling thread: commit callback, rebalance callback.
• rm: RequestManagers. e.g. Heartbeat, FindCoordinatorRequest.
• subscriptionState design is still under discussion

...

Application thread and its lifecycle

The polling thread handles API invocation and any responses from the background thread. Let's demonstrate its life cycle using the simple consumer use case (assign(), poll(), commitSync()) :

...

• UNJOINED: There's no rebalance. For the simple consumed use case, the GroupState remains in UNJOINED
• PREPARE: Sending the heartbeat and await the response
• ASSIGN: Assignment updated, client thread side callbacks are triggered, and await completion
• COMPLETE: Client thread callback completed and has notified the background thread.
• STABLE: stable group

Consumer group member state machine

It becomes clear when reading KIP-848 that the work of keeping the consumer group in proper state is fairly involved. We therefore turn our focus now to the logic needed for the consumer group member state machine (hereafter, CGMSM). 

...

The refactor should have (almost) no regression or breaking changes upon merge. So user should be able to continue using the new client. draw.io DiagrambordertruediagramNameApplication and network threadssimpleViewerfalsewidthlinksautotbstyletoplboxtruediagramWidth923revision1