Status

Current state: Under Discussion

Discussion thread:

JIRA:

Motivation

In production, producers commonly send keyed messages to leverage semantic partitioning for ordering guarantees, stream joins, and cross-cluster replication. However, kafka-producer-perf-test always produces records with null keys, making benchmark results systematically optimistic and unable to reflect the performance characteristics of real keyed workloads.

This proposal adds key distribution support to kafka-producer-perf-test, allowing engineers to benchmark keyed workloads with configurable key ranges and distribution strategies.

Public Interfaces

This proposal adds two new command-line arguments to kafka-producer-perf-test:

--key-distribution <none|range|random> (optional, default: none)

Controls how message keys are assigned:

none — null key (current behavior, default)
range — keys cycle through integers 0, 1, ..., KEY-RANGE-1 in round-robin order
random — each record gets a randomly selected integer from [0, KEY-RANGE)

--message-key-range <KEY-RANGE> (optional, required when --key-distribution is range or random)

Defines the size of the key space. Must be a positive integer.

Proposed Changes

New Enum: KeyDistribution

public enum KeyDistribution {
  NONE, RANGE, RANDOM
}

Generate key

Keys are serialized as their decimal string representation encoded in UTF-8, consistent with the ByteArraySerializer already configured for the producer. This keeps keys human-readable in tools like kafka-console-consumer.

Distribution	Key value
NONE	null
RANGE	Integer.toString(recordIndex % keyRange)
RANDOM	Integer.toString(random.nextInt(keyRange))

Validation

ConfigPostProcessor enforces mutual consistency between the two new arguments:

Condition	Error
--key-distribution range or random without --message-key-range	--message-key-range is required when --key-distribution is 'range' or 'random'.
--message-key-range specified with --key-distribution none	--key-distribution must be 'range' or 'random' when --message-key-range is specified.
--message-key-range ≤ 0	--message-key-range should be greater than zero.

Example Usage

Null keys — existing behavior (default)

bin/kafka-producer-perf-test.sh \
  --topic my-topic --num-records 1000000 --record-size 1024 \
  --throughput -1 --bootstrap-server localhost:9092

Round-robin across 100 distinct keys

bin/kafka-producer-perf-test.sh \
  --topic my-topic --num-records 1000000 --record-size 1024 \
  --throughput -1 --bootstrap-server localhost:9092 \
  --key-distribution range --message-key-range 100

Random keys from a space of 10,000

bin/kafka-producer-perf-test.sh \
  --topic my-topic --num-records 1000000 --record-size 1024 \
  --throughput -1 --bootstrap-server localhost:9092 \
  --key-distribution random --message-key-range 10000

Compatibility, Deprecation, and Migration Plan

The default value of --key-distribution is none, which preserves the current behavior of sending null-key records. Existing scripts and benchmarks continue to work without modification.

Test Plan

All remaining tests should pass, and new unit test.

Rejected Alternatives

n/a