Status

Current state: Under DiscussionDiscussion thread: hereAccepted

Discussion thread: https://lists.apache.org/thread/dvc6pp2h5oxo4bcq1mnfbs02z0g7pk61

Voting thread: https://lists.apache.org/thread/nqkv6ckppz5o6s9h1ls3qzhlmwchxqn1

JIRA: KAFKA-12392

Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).

...

The ConsoleProducer has two options, --batch-size and --max-partition-memory-bytes that , both of which configure the underlying batch.size producer config. One of them should be removed.

A previous previously discarded KIP-717 suggested removing --batch-size. However, this KIP argues for the opposite. Retaining --batch-size is more straightforward and aligns better with the terminology used throughout the Kafka 's community.

Public Interfaces

...

• kafka-console-producer.sh
  • The option Using the --max-partition-memory-bytes option will display a warning:
    "Warning: --max-partition-memory-bytes is deprecated and will be marked as deprecated and then removed in next major release.removed in a future version. Use --batch-size instead." 

Proposed Changes

...

• Correct the inaccurate description

...

• of --batch-size

...

• to reflect that it controls byte size, not message count
• Mark --max-partition-memory-bytes as deprecated, directing users to --batch-size instead.

• To maintain backward compatibility, the existing behavior where --max-partition-memory-bytes

...

• overrides --batch-size will be preserved until the deprecated option is removed.

Before:

 ./kafka-console-producer.sh
This tool helps to read data from standard input and publish it to Kafka.
Option                                   Description                            
------                                   -----------                            
--batch-size <Integer: size>             Number of messages to send in a single 
                                           batch if they are not being sent     
                                           synchronously. please note that this 
                                           option will be replaced if max-      
                                           partition-memory-bytes is also set   
                                           (default: 16384)                     
...

--max-partition-memory-bytes <Integer:   The buffer size allocated for a        
  memory in bytes per partition>           partition. When records are received 
                                           which are smaller than this size the 
                                           producer will attempt to             
                                           optimistically group them together   
                                           until this size is reached. This is  
                                           the option to control `batch.size`   
                                           in producer configs. (default: 16384)

...

 ./kafka-console-producer.sh
This tool helps to read data from standard input and publish it to Kafka.
Option                                   Description                            
------                                   -----------                            
--batch-size <Integer: size>             The buffer size in bytes allocated for a partition.
 										   When records are received 
                                           which are smaller than this size the 
                                           producer will attempt to             
                                           optimistically group them together   
                                           until this size is reached. This is  
                                           the option to control `batch.size`   
                                           in producer configs. (default: 16384)
             							   please note that this option will be 
...										   replaced if max-partition-memory-bytes
										   is also set.

--max-partition-memory-bytes <Integer:   (Deprecated) PleaseThe usebuffer --batch-size insteadsize allocated for a partition.
  memory memory in bytes per partition>           The buffer size allocated for a partition. 
										   When records are received 
                                                                  which are smaller than this size the 
                                           producer will attempt to             
                                           optimistically group them together   
                                           until this size is reached. This is  
                                           the option to control `batch.size`   
                                           in producer configs. (default: 16384)
							         	   This option will be removed in a 
										   future version. Use --batch-size instead.

./kafka-console-producer.sh --topic test --bootstrap-server localhost:9092 --max-partition-memory-bytes 16384
Warning: --max-partition-memory-bytes is deprecated and will be removed in a future version. Use --batch-size instead.

Compatibility, Deprecation, and Migration Plan

• Users should replace --max-partition-memory-bytes with --batch-size in their scripts.
• The deprecated option is planned for complete removal in a the next major release

Test Plan

N/A

Rejected Alternatives

...

Deprecate the --batch-size option

Use --max-memory-bytes to indicate batch.size in ProducerConfig is less intuitive.

...

Deprecate both --batch-size and --max-memory-bytes

...

in favor of generic properties

Encouraging users to set producer properties via the --command-config  or --command-property

...

  options.

This was rejected Rejected because keeping the dedicated --batch-size  flag is more convenient for users who frequently test the interaction between batch.size  and linger.ms.

Note: The recent KIP-1147 unified option names across many tools. It replaced --producer-property and --consumer-property with a consistent --command-property, and unified file-based options into --command-config.