You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 7 Next »

Overview

Engineers

@lianet Philip Nee Kirk True 

Approvers

Jason Gustafson guozhang Wang Rajini Sivaram

Objective

Refactor the current KafkaConsumer implementation to separate out the API processing in the user thread from the network I/O and heartbeat processing in a dedicated background thread

Status

In progress

Problem Statement

The current KafkaConsumer implementation suffers from a number of issues related to an inconsistent approach to ownership of resources among different threads. Complicated and error-prone synchronization and communication has been applied that makes the code difficult to reason on and has lead to a class of bugs and issues.

Scope

Must have

  • 100% API compatibility with existing KafkaConsumer

  • Test success rate parity with existing KafkaConsumer (i.e. no new test failures)

Nice to have

Not in scope

  • API-level changes (we don’t want to write a KIP)

References

Jira: Unable to render Jira issues macro, execution error.

bugs: https://issues.apache.org/jira/issues/?jql=labels%20%3D%20new-consumer-threading-should-fix

Subpages

  • No labels