...

This document highlights our effort to refactor the existing threading model of the KafkaConsumer, and the purpose of this refactor is to address issues and shortcoming shortcomings we’ve observed encountered in the past years, such as increasing code complexity from the hotfixes and patches, undesired concurrency behavior introduced by having both heartbeat thread and polling thread communicate to the coordinatorlock and race conditions caused by both heartbeatThread and the polling thread making coordinator requests, and the coupling between the client poll and the rebalance progress.

• Code complexity: the current rebalance protocol is quite heavy on the client side. And over the years, the patches have gotten increasingly complicated, and the readability and clarity has been impacted. For example, coordinator communication can happen at different places, that makes it difficult to understand the code path. There are also lots of small patches to handle bugs and edge cases that might further down require more fixes.

• Complex Coordinator Communication: To the previous point, as As the poling thread and the heartbeat thread both talk to the coordinator, it increases the code complexity and has introduced caused concurrent issues in the past.

• Asynchronous Background Thread: One of our goal here is to make rebalance process to happen asynchronously . And the advantages are, firstlyto the poll. Firstly, it simplifies the polling thread design, as all of the coordinator communication will be moved to the background. Secondly, it prevents polls from blocking the other consumer operations such as network requests.

Scope

The goal of this refactor, is to move We will be moving network communication, including heartbeat to the background thread, which allows the consumer to operate in a more asynchronous fashion. In particular, the scope of this refactor will be as such:

...