You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 2 Next »

Status

Current state: Under Discussion

Discussion thread: here

JIRA: KAFKA-12392

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

Motivation

The ConsoleProducer has two options, --batch-size and --max-partition-memory-bytes, both of which configure the underlying batch.size producer config. One of them should be removed.

A previously discarded KIP-717 suggested removing --batch-size . However, this KIP argues for the opposite. Retaining --batch-size  is more straightforward and aligns better with the terminology used throughout the Kafka community.

Public Interfaces

This change affects the command-line interface of 

  • kafka-console-producer.sh
    • The option --max-partition-memory-bytes will be marked as deprecated and then removed in the next major release.

Proposed Changes

  • Correct the inaccurate description of --batch-size to reflect that it controls byte size, not message count
  • Mark --max-partition-memory-bytes as deprecated, directing users to --batch-size instead.

  • To maintain backward compatibility, the existing behavior where --max-partition-memory-bytes  overrides --batch-size  will be preserved until the deprecated option is removed.

Before:

 ./kafka-console-producer.sh
This tool helps to read data from standard input and publish it to Kafka.
Option                                   Description                            
------                                   -----------                            
--batch-size <Integer: size>             Number of messages to send in a single 
                                           batch if they are not being sent     
                                           synchronously. please note that this 
                                           option will be replaced if max-      
                                           partition-memory-bytes is also set   
                                           (default: 16384)                     
...

--max-partition-memory-bytes <Integer:   The buffer size allocated for a        
  memory in bytes per partition>           partition. When records are received 
                                           which are smaller than this size the 
                                           producer will attempt to             
                                           optimistically group them together   
                                           until this size is reached. This is  
                                           the option to control `batch.size`   
                                           in producer configs. (default: 16384)

After:

 ./kafka-console-producer.sh
This tool helps to read data from standard input and publish it to Kafka.
Option                                   Description                            
------                                   -----------                            
--batch-size <Integer: size>             The buffer size in bytes allocated for a partition.
 										   When records are received 
                                           which are smaller than this size the 
                                           producer will attempt to             
                                           optimistically group them together   
                                           until this size is reached. This is  
                                           the option to control `batch.size`   
                                           in producer configs. (default: 16384)
             							   please note that this 
                                           option will be replaced if max-      
                                           partition-memory-bytes is also set.

--max-partition-memory-bytes <Integer:   (Deprecated) Please use --batch-size instead.
  memory in bytes per partition>           The buffer size allocated for a partition. 
										   When records are received 
                                           which are smaller than this size the 
                                           producer will attempt to             
                                           optimistically group them together   
                                           until this size is reached. This is  
                                           the option to control `batch.size`   
                                           in producer configs. (default: 16384)

Compatibility, Deprecation, and Migration Plan

  • Users should replace --max-partition-memory-bytes with --batch-size in their scripts.
  • The deprecated option is planned for complete removal in the next major release

Test Plan

N/A

Rejected Alternatives

Deprecating the --batch-size option

Use --max-memory-bytes to indicate batch.size in ProducerConfig is less intuitive.

Deprecating both --batch-size and --max-memory-bytes in favor of generic properties

Encouraging users to set producer properties via the --command-config  or --command-property  options.

This was rejected because keeping the dedicated --batch-size flag is more convenient for users who frequently test the interaction between batch.size  and linger.ms .

Note: The recent KIP-1147 unified option names across many tools. It replaced --producer-property and --consumer-property with a consistent --command-property, and unified file-based options into --command-config.

  • No labels