You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 8 Next »

This page is meant as a template for writing a KIP. To create a KIP choose Tools->Copy on this page and modify with your content and replace the heading with the next KIP number and a description of your issue. Replace anything in italics with your own description.

Status

Current state: "Under Discussion"

Discussion thread: here

JIRA: here

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

KafkaProducer is designed to be thread-safe and we encourage users to share a single producer instance across multiple threads[link]. These threads often need to send records to different topics. However, another important configuration—acks—is currently set at the producer level only. As a result, users needing different acks settings for each topic must create additional producer instances instead of reusing the existing one. This goes against our original intent and prevents users from reusing KafkaProducer instance.

In this KIP, we propose adding topic-level acks, enabling users reuse a single KafkaProducer even when different topics require different acks settings. This change will reduce the need for multiple producers, which is particularly beneficial in resource-constrained environments.

Public Interfaces

We add a new config TOPIC_ACKS_CONFIG  and TOPIC_ACKS_DOC into ProducerConfig and define the format.

Format: acks.topic=acks.<TopicA>=<acks1>:acks.<TopicB>=<acks2>

Name

Type

Importance

Default

Description

acks.topicStringLOW
null

This configuration item will set acks for specific topics.

Proposed Changes

  • Adding a new configuration in  ProducerConfig : acks.topic,  By setting this configuration item, users can customize the acks for specific topic.
  • Behavior change:
    1. Attach topic-level acks to RecordAccumulator#TopicInfo.
    2. Return Map<Acks, List<ProducerBatch>> when RecordAccumulator#drainBatchesForOneNode is called.
    3. Finally, we can get the acks information and group same acks into List<ProducerBatch>> for a node in sender#sendProduceRequests and then send request.

Compatibility, Deprecation, and Migration Plan

  • The public interface is new one so there are no compatibility issues.
  • The ProduceRequest already contains an acks field so we don't need add any additional change at PRC-level [link].

Test Plan

  • Relevant unit tests and integration tests will be added to demonstrate the functionality.

Rejected Alternatives

  • N/A
  • No labels