You are viewing an old version of this page. View the current version.

Compare with Current View Page History

Version 1 Next »

Status

Current state: Under Discussion

Discussion thread: here

JIRA: KAFKA-12392

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

The ConsoleProducer has two options, --batch-size and --max-partition-memory-bytes that both configure the underlying batch.size producer config. One of them should be removed.

A previous discarded KIP-717 suggested removing --batch-size . However, this KIP argues for the opposite. Retaining --batch-size  is more straightforward and aligns better with the terminology used throughout Kafka's community.

Public Interfaces

This change affects the command-line interface of 

  • kafka-console-producer.sh
    • The option --max-memory-bytes will be marked as deprecated and then removed in next major release.

Proposed Changes

Fix the description for --batch-size and add deprecated message for option --max-partition-memory-bytes 

Before:

 ./kafka-console-producer.sh
This tool helps to read data from standard input and publish it to Kafka.
Option                                   Description                            
------                                   -----------                            
--batch-size <Integer: size>             Number of messages to send in a single 
                                           batch if they are not being sent     
                                           synchronously. please note that this 
                                           option will be replaced if max-      
                                           partition-memory-bytes is also set   
                                           (default: 16384)                     
...

--max-partition-memory-bytes <Integer:   The buffer size allocated for a        
  memory in bytes per partition>           partition. When records are received 
                                           which are smaller than this size the 
                                           producer will attempt to             
                                           optimistically group them together   
                                           until this size is reached. This is  
                                           the option to control `batch.size`   
                                           in producer configs. (default: 16384)

After:

 ./kafka-console-producer.sh
This tool helps to read data from standard input and publish it to Kafka.
Option                                   Description                            
------                                   -----------                            
--batch-size <Integer: size>             The buffer size in bytes allocated for a partition.
 										   When records are received 
                                           which are smaller than this size the 
                                           producer will attempt to             
                                           optimistically group them together   
                                           until this size is reached. This is  
                                           the option to control `batch.size`   
                                           in producer configs. (default: 16384)          
...

--max-partition-memory-bytes <Integer:   (Deprecated) Please use --batch-size instead.
  memory in bytes per partition>           The buffer size allocated for a partition. 
										   When records are received 
                                           which are smaller than this size the 
                                           producer will attempt to             
                                           optimistically group them together   
                                           until this size is reached. This is  
                                           the option to control `batch.size`   
                                           in producer configs. (default: 16384)

Compatibility, Deprecation, and Migration Plan

  • Users should replace --max-partition-memory-bytes with --batch-size in their scripts.
  • The deprecated option is planned for complete removal in a next major release

Test Plan

N/A

Rejected Alternatives

Deprecated option --batch-size

Use --max-memory-bytes to indicate batch.size in ProducerConfig is less intuitive.


Deprecated both --batch-size and --max-memory-bytes. Instead, use --command-config or --command-property to pass producer properties

Rejected because keeping the dedicated --batch-size  flag is more convenient for users who frequently test the interaction between batch.size and linger.ms.




  • No labels