Status

Current state: Under Discussion

Discussion thread:

JIRA:

Motivation

kafka-producer-perf-test is the standard tool for benchmarking Kafka producer throughput and latency. However, it currently always creates records with a null key, which means all messages are distributed across partitions via round-robin.

In production workloads, producers commonly send messages with keys to leverage semantic partitioning — ensuring that records with the same key always land on the same partition.

Public Interfaces

This proposal adds two new command-line arguments to kafka-producer-perf-test:

--key-distribution <none|range|random> (optional, default: none)

Controls how message keys are assigned:

none — null key (current behavior, default)
range — keys cycle through integers 0, 1, ..., KEY-RANGE-1 in round-robin order
random — each record gets a randomly selected integer from [0, KEY-RANGE)

--message-key-range <KEY-RANGE> (optional, required when --key-distribution is range or random)

Defines the size of the key space. Must be a positive integer.

Proposed Changes

New Enum: KeyDistribution

public enum KeyDistribution {
  NONE, RANGE, RANDOM
}

Generate key

Keys are serialized as their decimal string representation encoded in UTF-8, consistent with the ByteArraySerializer already configured for the producer. This keeps keys human-readable in tools like kafka-console-consumer.

Distribution	Key value
NONE	null
RANGE	Integer.toString(recordIndex % keyRange)
RANDOM	Integer.toString(random.nextInt(keyRange))

Validation

ConfigPostProcessor enforces mutual consistency between the two new arguments:

Condition	Error
--key-distribution range or random without --message-key-range	--message-key-range is required when --key-distribution is 'range' or 'random'.
--message-key-range specified with --key-distribution none	--key-distribution must be 'range' or 'random' when --message-key-range is specified.
--message-key-range ≤ 0	--message-key-range should be greater than zero.

Example Usage

Null keys — existing behavior (default)

bin/kafka-producer-perf-test.sh \
  --topic my-topic --num-records 1000000 --record-size 1024 \
  --throughput -1 --bootstrap-server localhost:9092

Round-robin across 100 distinct keys

bin/kafka-producer-perf-test.sh \
  --topic my-topic --num-records 1000000 --record-size 1024 \
  --throughput -1 --bootstrap-server localhost:9092 \
  --key-distribution range --message-key-range 100

Random keys from a space of 10,000

bin/kafka-producer-perf-test.sh \
  --topic my-topic --num-records 1000000 --record-size 1024 \
  --throughput -1 --bootstrap-server localhost:9092 \
  --key-distribution random --message-key-range 10000

Compatibility, Deprecation, and Migration Plan

The default value of --key-distribution is none, which preserves the current behavior of sending null-key records. Existing scripts and benchmarks continue to work without modification.

Test Plan

All remaining tests should pass, and new unit test.

Rejected Alternatives

n/a