Status

Current state: Under Discussion

...

JIRA:

Motivation

In production, producers commonly send keyed messages to leverage semantic partitioning for ordering guarantees, stream joins, and cross-cluster replication. However, kafka-producer-perf-test always produces records with null keys, making benchmark results systematically optimistic and unable to reflect the performance characteristics of real keyed workloads.

This proposal adds key distribution support to kafka-producer-perf-test, allowing engineers to benchmark keyed workloads with configurable key ranges and distribution strategies.

Public Interfaces

This proposal adds two new command-line arguments to kafka-producer-perf-test:

--key-distribution <none|range|random> (optional, default: none)

Controls how message keys are assigned:

• none — null key (current behavior, default)
• range — keys cycle through integers 0, 1, ..., KEY-RANGE-1 in round-robin order
• random — each record gets a randomly selected integer from [0, KEY-RANGE)

--message-key-range <KEY-RANGE> (optional, required when --key-distribution is range or random)

Defines the size of the key space. Must be a positive integer.

Proposed Changes

New Enum: KeyDistribution

public enum KeyDistribution {
  NONE, RANGE, RANDOM
}

Generate key

Keys are serialized as their decimal string representation encoded in UTF-8, consistent with the ByteArraySerializer already configured for the producer. This keeps keys human-readable in tools like kafka-console-consumer.

...

Performance note: The random distribution reuses a single SplittableRandom instance that is already constructed for payload generation. SplittableRandom.nextInt() is a lightweight, non-thread-safe PRNG with no allocation overhead, so key generation adds negligible latency to the hot path.

Validation

ConfigPostProcessor enforces mutual consistency between the two new arguments:

Example Usage

• Null keys — existing behavior (default)
  

  Code Block
  bin/kafka-producer-perf-test.sh \ --topic my-topic --num-records 1000000 --record-size 1024 \ --throughput -1 --bootstrap-server localhost:9092

• Round-robin across 100 distinct keys
  

  Code Block
  bin/kafka-producer-perf-test.sh \ --topic my-topic --num-records 1000000 --record-size 1024 \ --throughput -1 --bootstrap-server localhost:9092 \ --key-distribution range --message-key-range 100

• Random keys from a space of 10,000
  

  Code Block
  bin/kafka-producer-perf-test.sh \ --topic my-topic --num-records 1000000 --record-size 1024 \ --throughput -1 --bootstrap-server localhost:9092 \ --key-distribution random --message-key-range 10000

Compatibility, Deprecation, and Migration Plan

The default value of --key-distribution is none, which preserves the current behavior of sending null-key records. Existing scripts and benchmarks continue to work without modification.

Test Plan

All remaining tests should pass, and new unit test.

Rejected Alternatives

UUID keys for random distribution

An alternative design would use UUID.randomUUID().toString()  as the key for the random distribution, providing globally unique keys with no repeated values across the entire benchmark run.

...