You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 2 Next »

Status

Current state: Under Discussion

Discussion thread: 

JIRA: KAFKA-967 - Getting issue details... STATUS

Motivation

In production, producers commonly send keyed messages to leverage semantic partitioning for ordering guarantees, stream joins, and cross-cluster replication. However, kafka-producer-perf-test always produces records with null keys, making benchmark results systematically optimistic and unable to reflect the performance characteristics of real keyed workloads.

This proposal adds key distribution support to kafka-producer-perf-test, allowing engineers to benchmark keyed workloads with configurable key ranges and distribution strategies.

Public Interfaces

This proposal adds two new command-line arguments to kafka-producer-perf-test:

--key-distribution <none|range|random> (optional, default: none)

Controls how message keys are assigned:

  • none — null key (current behavior, default)
  • range — keys cycle through integers 0, 1, ..., KEY-RANGE-1 in round-robin order
  • random — each record gets a randomly selected integer from [0, KEY-RANGE)

--message-key-range <KEY-RANGE> (optional, required when --key-distribution is range or random)

Defines the size of the key space. Must be a positive integer.

Proposed Changes

New Enum: KeyDistribution

public enum KeyDistribution {
  NONE, RANGE, RANDOM
}

Generate key

Keys are serialized as their decimal string representation encoded in UTF-8, consistent with the ByteArraySerializer already configured for the producer. This keeps keys human-readable in tools like kafka-console-consumer.

DistributionKey value
NONEnull
RANGEInteger.toString(recordIndex % keyRange)  
RANDOM
Integer.toString(random.nextInt(keyRange))

Validation

ConfigPostProcessor enforces mutual consistency between the two new arguments:

ConditionError
--key-distribution range or random without --message-key-range--message-key-range is required when --key-distribution is 'range' or 'random'.
--message-key-range specified with --key-distribution none--key-distribution must be 'range' or 'random' when --message-key-range is specified.
--message-key-range ≤ 0--message-key-range should be greater than zero.

Example Usage

  • Null keys — existing behavior (default)
    bin/kafka-producer-perf-test.sh \
  --topic my-topic --num-records 1000000 --record-size 1024 \
  --throughput -1 --bootstrap-server localhost:9092
  • Round-robin across 100 distinct keys
    bin/kafka-producer-perf-test.sh \
  --topic my-topic --num-records 1000000 --record-size 1024 \
  --throughput -1 --bootstrap-server localhost:9092 \
  --key-distribution range --message-key-range 100
  • Random keys from a space of 10,000
    bin/kafka-producer-perf-test.sh \
  --topic my-topic --num-records 1000000 --record-size 1024 \
  --throughput -1 --bootstrap-server localhost:9092 \
  --key-distribution random --message-key-range 10000

Compatibility, Deprecation, and Migration Plan

The default value of --key-distribution is none, which preserves the current behavior of sending null-key records. Existing scripts and benchmarks continue to work without modification.

Test Plan

All remaining tests should pass, and new unit test.

Rejected Alternatives

n/a

  • No labels