Status

Current state: Under Discussion Accepted

Discussion thread: https://lists.apache.org/thread/vdp8scrrzdq7ofvl0mm84dhphq8kmzgc

...

In production, producers commonly send keyed messages records to leverage semantic partitioning for ordering guarantees, stream joins, and cross-cluster replication. However, kafka-producer-perf-test always produces records with null keys, making benchmark results systematically optimistic and unable to reflect the performance characteristics of real keyed workloads. Additionally, since log compaction requires non-null keys, the tool currently cannot benchmark compacted topics at all. Adding key support removes this limitation.

This proposal adds key distribution support to kafka-producer-perf-test, allowing engineers to benchmark keyed workloads with configurable key ranges and distribution strategies.

...

• none — establishes a null-key baseline for topics where ordering and co-partitioning are not required, such as log aggregation pipelines.
• range — models workloads where a bounded, predictable set of keys cycles repeatedly, such as Kafka Streams joins or MirrorMaker 2 replication, where the same key must consistently land on the same partition to preserve ordering guarantees.
• random — models workloads with a bounded but unpredictably distributed key space, where keys arrive in non-deterministic order rather than cycling sequentially. A large range (e.g., 1,000,000) approximates unique-key workloads such as IoT device data without the overhead of UUID generation.

Note: that both range and random produce a uniform key distribution. Skewed distributions are out of scope for this proposal and may be addressed in a future KIP.

Public Interfaces

This proposal adds two new command-line arguments to kafka-producer-perf-test:

--key-distribution <none|range|random> (optional, default: none)

Controls how message record keys are assigned:

• none — null key (current behavior, default)
• range — keys cycle through integers 0, 1, ..., KEY-RANGE-1 in round-robin order
• random — each record gets a randomly selected integer from [0, KEY-RANGE)

--

...

record-key-range <KEY-RANGE> (optional, required when --key-distribution is range or random)

Defines the size of the key space. Must be a positive integer.

...

...

• Null keys — existing behavior (default)
  

  Code Block
  bin/kafka-producer-perf-test.sh \ --topic my-topic --num-records 1000000 --record-size 1024 \ --throughput -1 --bootstrap-server localhost:9092

• Round-robin across 100 distinct keys
  

  Code Block
  bin/kafka-producer-perf-test.sh \ --topic my-topic --num-records 1000000 --record-size 1024 \ --throughput -1 --bootstrap-server localhost:9092 \ --key-distribution range --messagerecord-key-range 100

• Random keys from a space of 10,000
  

  Code Block
  bin/kafka-producer-perf-test.sh \ --topic my-topic --num-records 1000000 --record-size 1024 \ --throughput -1 --bootstrap-server localhost:9092 \ --key-distribution random --messagerecord-key-range 10000

Compatibility, Deprecation, and Migration Plan

...

Engineers who genuinely need globally unique keys can use --key-distribution random --messagerecord-key-range <large-number> (e.g., 2^31−1) to approximate the same effect without the overhead.