Current state: Under Discussion
Discussion thread: https://lists.apache.org/thread/dvc6pp2h5oxo4bcq1mnfbs02z0g7pk61
Voting thread: https://lists.apache.org/thread/nqkv6ckppz5o6s9h1ls3qzhlmwchxqn1
JIRA: KAFKA-12392
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
The ConsoleProducer has two options, --batch-size and --max-partition-memory-bytes, both of which configure the underlying batch.size producer config. One of them should be removed.
A previously discarded KIP-717 suggested removing --batch-size. However, this KIP argues for the opposite. Retaining --batch-size is more straightforward and aligns better with the terminology used throughout the Kafka community.
This change affects the command-line interface of
kafka-console-producer.sh--max-partition-memory-bytes option will display a warning:"Warning: --max-partition-memory-bytes is deprecated and will be removed in a future version. Use --batch-size instead." --batch-size to reflect that it controls byte size, not message count--max-partition-memory-bytes as deprecated, directing users to --batch-size instead.To maintain backward compatibility, the existing behavior where --max-partition-memory-bytes overrides --batch-size will be preserved until the deprecated option is removed.
./kafka-console-producer.sh
This tool helps to read data from standard input and publish it to Kafka.
Option Description
------ -----------
--batch-size <Integer: size> Number of messages to send in a single
batch if they are not being sent
synchronously. please note that this
option will be replaced if max-
partition-memory-bytes is also set
(default: 16384)
...
--max-partition-memory-bytes <Integer: The buffer size allocated for a
memory in bytes per partition> partition. When records are received
which are smaller than this size the
producer will attempt to
optimistically group them together
until this size is reached. This is
the option to control `batch.size`
in producer configs. (default: 16384) |
./kafka-console-producer.sh
This tool helps to read data from standard input and publish it to Kafka.
Option Description
------ -----------
--batch-size <Integer: size> The buffer size in bytes allocated for a partition.
When records are received
which are smaller than this size the
producer will attempt to
optimistically group them together
until this size is reached. This is
the option to control `batch.size`
in producer configs. (default: 16384)
please note that this option will be
replaced if max-partition-memory-bytes
is also set.
--max-partition-memory-bytes <Integer: (Deprecated) The buffer size allocated for a partition.
memory in bytes per partition> When records are received
which are smaller than this size the
producer will attempt to
optimistically group them together
until this size is reached. This is
the option to control `batch.size`
in producer configs. (default: 16384)
This option will be removed in a
future version. Use --batch-size instead.
./kafka-console-producer.sh --topic test --bootstrap-server localhost:9092 --max-partition-memory-bytes 16384
Warning: --max-partition-memory-bytes is deprecated and will be removed in a future version. Use --batch-size instead. |
--max-partition-memory-bytes with --batch-size in their scripts.N/A
--batch-size optionUse --max-memory-bytes to indicate batch.size in ProducerConfig is less intuitive.
--max-memory-bytes in favor of generic propertiesEncouraging users to set producer properties via the --command-config or --command-property options.
This was rejected because keeping the dedicated --batch-size flag is more convenient for users who frequently test the interaction between batch.size and linger.ms.
Note: The recent KIP-1147 unified option names across many tools. It replaced --producer-property and --consumer-property with a consistent --command-property, and unified file-based options into --command-config.