You are viewing an old version of this page. View the current version.

Compare with Current View Page History

Version 1 Next »

Status

Current state: Under Discussion

Discussion thread: 

JIRA: KAFKA-967 - Getting issue details... STATUS

Motivation

kafka-producer-perf-test is the standard tool for benchmarking Kafka producer throughput and latency. However, it currently always creates records with a null key, which means all messages are distributed across partitions via round-robin.

In production workloads, producers commonly send messages with keys to leverage semantic partitioning — ensuring that records with the same key always land on the same partition.


Public Interfaces

This proposal adds two new command-line arguments to kafka-producer-perf-test:

--key-distribution <none|range|random> (optional, default: none)

Controls how message keys are assigned:

  • none — null key (current behavior, default)
  • range — keys cycle through integers 0, 1, ..., KEY-RANGE-1 in round-robin order
  • random — each record gets a randomly selected integer from [0, KEY-RANGE)

--message-key-range <KEY-RANGE> (optional, required when --key-distribution is range or random)

Defines the size of the key space. Must be a positive integer.

Proposed Changes

New Enum: KeyDistribution

public enum KeyDistribution {
  NONE, RANGE, RANDOM
}

Generate key

Keys are serialized as their decimal string representation encoded in UTF-8, consistent with the ByteArraySerializer already configured for the producer. This keeps keys human-readable in tools like kafka-console-consumer.

DistributionKey value
NONEnull
RANGEInteger.toString(recordIndex % keyRange)  
RANDOM
Integer.toString(random.nextInt(keyRange))

Validation

ConfigPostProcessor enforces mutual consistency between the two new arguments:

ConditionError
--key-distribution range or random without --message-key-range--message-key-range is required when --key-distribution is 'range' or 'random'.
--message-key-range specified with --key-distribution none--key-distribution must be 'range' or 'random' when --message-key-range is specified.
--message-key-range ≤ 0--message-key-range should be greater than zero.

Example Usage

  • Null keys — existing behavior (default)
    bin/kafka-producer-perf-test.sh \
  --topic my-topic --num-records 1000000 --record-size 1024 \
  --throughput -1 --bootstrap-server localhost:9092
  • Round-robin across 100 distinct keys
    bin/kafka-producer-perf-test.sh \
  --topic my-topic --num-records 1000000 --record-size 1024 \
  --throughput -1 --bootstrap-server localhost:9092 \
  --key-distribution range --message-key-range 100
  • Random keys from a space of 10,000
    bin/kafka-producer-perf-test.sh \
  --topic my-topic --num-records 1000000 --record-size 1024 \
  --throughput -1 --bootstrap-server localhost:9092 \
  --key-distribution random --message-key-range 10000

Compatibility, Deprecation, and Migration Plan

The default value of --key-distribution is none, which preserves the current behavior of sending null-key records. Existing scripts and benchmarks continue to work without modification.

Test Plan

All remaining tests should pass, and new unit test.

Rejected Alternatives

n/a

  • No labels