Versions Compared

Old Version 3

changes.mady.by.user Kirk True

Saved on

compared with

New Version 4

changes.mady.by.user Kirk True

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Table of Contents

 Overview

Engineers

@lianet Philip Nee Kirk True 

Approvers

Jason Gustafson guozhang Wang Rajini Sivaram

Objective

Refactor the current KafkaConsumer implementation to separate out the API processing in the user thread from the network I/O and heartbeat processing in a dedicated background thread

Status

In progress

Problem Statement

The current KafkaConsumer implementation suffers from a number of issues related to an inconsistent approach to ownership of resources among different threads. Complicated and error-prone synchronization and communication has been applied that makes the code difficult to reason on and has lead to a class of bugs and issues.

Scope

Must have

  • 100% API compatibility with existing KafkaConsumer

  • Test success rate parity with existing KafkaConsumer (i.e. no new test failures)

Nice to have

Not in scope

  • API-level changes (we don’t want to write a KIP)

Reference materials

Design: Proposal: Consumer Threading Model Refactor

Main Jira:

Jira
serverASF JIRA
serverId5aa69414-a9e9-3523-82ec-879b028fb15b
keyKAFKA-14246

bugs: https://issues.apache.org/jira/issues/?jql=labels%20%3D%20new-consumer-threading-should-fix