Status

Current state: Accepted

Discussion thread: https://lists.apache.org/thread/vdp8scrrzdq7ofvl0mm84dhphq8kmzgc

Vote thread: https://lists.apache.org/thread/5280h24g205vn69dxr44lc15dt3ncrvz

JIRA: KAFKA-967 - Getting issue details... STATUS

Motivation

In production, producers commonly send keyed records to leverage semantic partitioning for ordering guarantees, stream joins, and cross-cluster replication. However, kafka-producer-perf-test always produces records with null keys, making benchmark results systematically optimistic and unable to reflect the performance characteristics of real keyed workloads. Additionally, since log compaction requires non-null keys, the tool currently cannot benchmark compacted topics at all. Adding key support removes this limitation.

This proposal adds key distribution support to kafka-producer-perf-test, allowing engineers to benchmark keyed workloads with configurable key ranges and distribution strategies.

The three distribution modes cover the most common real-world keyed workload patterns:

• none — establishes a null-key baseline for topics where ordering and co-partitioning are not required, such as log aggregation pipelines.
• range — models workloads where a bounded, predictable set of keys cycles repeatedly, such as Kafka Streams joins or MirrorMaker 2 replication, where the same key must consistently land on the same partition to preserve ordering guarantees.
• random — models workloads with a bounded but unpredictably distributed key space, where keys arrive in non-deterministic order rather than cycling sequentially. A large range (e.g., 1,000,000) approximates unique-key workloads such as IoT device data without the overhead of UUID generation.

Note: that both range and random produce a uniform key distribution. Skewed distributions are out of scope for this proposal and may be addressed in a future KIP.

Public Interfaces

This proposal adds two new command-line arguments to kafka-producer-perf-test:

--key-distribution <none|range|random> (optional, default: none)

Controls how record keys are assigned:

• none — null key (current behavior, default)
• range — keys cycle through integers 0, 1, ..., KEY-RANGE-1 in round-robin order
• random — each record gets a randomly selected integer from [0, KEY-RANGE)

--record-key-range <KEY-RANGE> (optional, required when --key-distribution is range or random)

Defines the size of the key space. Must be a positive integer.

--random-seed <seed> (optional, default 0)

Controls the seed for the pseudo-random number generator used by --key-distribution random and random payload generation. The default value of 0 ensures deterministic, reproducible benchmark runs. Set to a different value when non-repeating sequences are required.

Proposed Changes

New Enum: KeyDistribution

public enum KeyDistribution {
  NONE, RANGE, RANDOM
}

Generate key

Keys are serialized as their decimal string representation encoded in UTF-8, consistent with the ByteArraySerializer already configured for the producer. This keeps keys human-readable in tools like kafka-console-consumer.

RANDOM

Performance note: The random distribution reuses a single SplittableRandom instance that is already constructed for payload generation. SplittableRandom.nextInt() is a lightweight, non-thread-safe PRNG with no allocation overhead, so key generation adds negligible latency to the hot path.

Validation

ConfigPostProcessor enforces mutual consistency between the two new arguments:

Example Usage

• Null keys — existing behavior (default)
  

  bin/kafka-producer-perf-test.sh \
    --topic my-topic --num-records 1000000 --record-size 1024 \
    --throughput -1 --bootstrap-server localhost:9092

• Round-robin across 100 distinct keys
  

  bin/kafka-producer-perf-test.sh \
    --topic my-topic --num-records 1000000 --record-size 1024 \
    --throughput -1 --bootstrap-server localhost:9092 \
    --key-distribution range --record-key-range 100

• Random keys from a space of 10,000
  

  bin/kafka-producer-perf-test.sh \
    --topic my-topic --num-records 1000000 --record-size 1024 \
    --throughput -1 --bootstrap-server localhost:9092 \
    --key-distribution random --record-key-range 10000

Compatibility, Deprecation, and Migration Plan

The default value of --key-distribution is none, which preserves the current behavior of sending null-key records. Existing scripts and benchmarks continue to work without modification.

Test Plan

All remaining tests should pass, and new unit test.

Rejected Alternatives

UUID keys for random distribution

An alternative design would use UUID.randomUUID().toString()  as the key for the random distribution, providing globally unique keys with no repeated values across the entire benchmark run.

This was rejected for two reasons:

1. Unbounded key space defeats the purpose. The primary use case for random keys is to benchmark workloads with a known, bounded key space (e.g., 10,000 customer IDs). UUID keys give every record a unique key, making partition distribution identical to round-robin and eliminating the ability to model hot-key or skewed-partition scenarios.
2. Performance overhead. UUID.randomUUID()  uses SecureRandom internally, which is significantly slower than SplittableRandom.nextInt()  and could become a bottleneck in high-throughput benchmarks — the opposite of what a perf tool should do.

Engineers who genuinely need globally unique keys can use --key-distribution random --record-key-range <large-number> (e.g., 2^31−1) to approximate the same effect without the overhead.