Status

Current state: Under Discussion

Discussion thread: 

JIRA: KAFKA-967 - Getting issue details... STATUS

Motivation

In production, producers commonly send keyed messages to leverage semantic partitioning for ordering guarantees, stream joins, and cross-cluster replication. However, kafka-producer-perf-test always produces records with null keys, making benchmark results systematically optimistic and unable to reflect the performance characteristics of real keyed workloads.

This proposal adds key distribution support to kafka-producer-perf-test, allowing engineers to benchmark keyed workloads with configurable key ranges and distribution strategies.

Public Interfaces

This proposal adds two new command-line arguments to kafka-producer-perf-test:

--key-distribution <none|range|random> (optional, default: none)

Controls how message keys are assigned:

• none — null key (current behavior, default)
• range — keys cycle through integers 0, 1, ..., KEY-RANGE-1 in round-robin order
• random — each record gets a randomly selected integer from [0, KEY-RANGE)

--message-key-range <KEY-RANGE> (optional, required when --key-distribution is range or random)

Defines the size of the key space. Must be a positive integer.

Proposed Changes

New Enum: KeyDistribution

public enum KeyDistribution {
  NONE, RANGE, RANDOM
}

Generate key

Keys are serialized as their decimal string representation encoded in UTF-8, consistent with the ByteArraySerializer already configured for the producer. This keeps keys human-readable in tools like kafka-console-consumer.

RANDOM

Performance note: The random distribution reuses a single SplittableRandom instance that is already constructed for payload generation. SplittableRandom.nextInt() is a lightweight, non-thread-safe PRNG with no allocation overhead, so key generation adds negligible latency to the hot path.

Validation

ConfigPostProcessor enforces mutual consistency between the two new arguments:

Example Usage

• Null keys — existing behavior (default)
  

  bin/kafka-producer-perf-test.sh \
    --topic my-topic --num-records 1000000 --record-size 1024 \
    --throughput -1 --bootstrap-server localhost:9092

• Round-robin across 100 distinct keys
  

  bin/kafka-producer-perf-test.sh \
    --topic my-topic --num-records 1000000 --record-size 1024 \
    --throughput -1 --bootstrap-server localhost:9092 \
    --key-distribution range --message-key-range 100

• Random keys from a space of 10,000
  

  bin/kafka-producer-perf-test.sh \
    --topic my-topic --num-records 1000000 --record-size 1024 \
    --throughput -1 --bootstrap-server localhost:9092 \
    --key-distribution random --message-key-range 10000

Compatibility, Deprecation, and Migration Plan

The default value of --key-distribution is none, which preserves the current behavior of sending null-key records. Existing scripts and benchmarks continue to work without modification.

Test Plan

All remaining tests should pass, and new unit test.

Rejected Alternatives

UUID keys for random distribution

An alternative design would use UUID.randomUUID().toString()  as the key for the random distribution, providing globally unique keys with no repeated values across the entire benchmark run.

This was rejected for two reasons:

1. Unbounded key space defeats the purpose. The primary use case for random keys is to benchmark workloads with a known, bounded key space (e.g., 10,000 customer IDs). UUID keys give every record a unique key, making partition distribution identical to round-robin and eliminating the ability to model hot-key or skewed-partition scenarios.
2. Performance overhead. UUID.randomUUID()  uses SecureRandom internally, which is significantly slower than SplittableRandom.nextInt()  and could become a bottleneck in high-throughput benchmarks — the opposite of what a perf tool should do.

Engineers who genuinely need globally unique keys can use --key-distribution random --message-key-range <large-number> (e.g., 2^31−1) to approximate the same effect without the overhead.