...

We propose a modification of the producer performance test to add a command line argument for "warmup records". The total number of records for the test must be strictly greater than the number of warmup records or the test should throw an error. Negative values for warmup records should also throw an error.  A value of "0" for warmup records should have the test behave identically to not specifying the parameter. Messages sent during warmup would be accumulated in a separate Stats object from the "post-warmup" records that make up the bulk steady state portion of the test. The warmup measurements would then be combined .

For reporting, there are two options that seem to come down to user preference. Both options have two summary lines, one containing warmup data and the other being purely steady state results.  In the first option, the warmup measurements could be summarized independently in a "warmup-only summary", followed by a summary of the steady state performance. With this option, the user would be able to understand performance during the warmup as a distinct phase of the measurement, then compare that to the steady state performance. The second option would be for the first summary print to combine the warmup data with the post-warmup latency measurements for a "full test summary" that is, effectively, the same output that the test currently has. This option allows users to compare what the test would have reported in the absence of a warmup and compare that to steady state performance. The first approach identifies warmup as a distinct phase, the second might be a preferred option to allow users to transition from their traditional use of the benchmark to the warmup-enabled version of the benchmark. This second option is our preferred implementation, but the behavior could be toggled with another command line option such as '--combined-summary'.

Regarding the process of choosing a warmup duration, the warmup records should comprise the time required for producers to startup, allocate resources, connect to the brokers, then reach steady state performance. Although a one-minute warmup is common in Java performance analysis, a simple process for more accurately determining the warmup time of a given topology would be to run the test without a warmup first. The resulting time-series of the producer performance test output would typically have high latency during the first few windows of output (at 5 s per window). The minimum warmup records could then be chosen to be the number of records sent once producer latency reaches an acceptable steady state. Although the producer performance test output should provide sufficient information to set a warmup, additional data sources such as JMX:producer:record-queue-time, JMX:producer:request-latency-avg, and the various broker queue-time metrics could be consulted to help precisely determine when steady state has been achieved. Since the warmup data will be reported separately from the desired steady state results, and running longer warmups should not impact steady state performance, it's always good practice to run a warmup for slightly longer than is deemed necessary to guarantee records from the warmup have been processed and ensure steady state operation.

As of this writing, the producer performance test seeks to meet its target throughput by evaluating the total time elapsed and total records sent since the producer started up. This could lead to higher steady-state producer throughput than expected if the producer is sending slowly during the warmup phase. This behavior may be a desired artifact of the producer's operation in some test cases, but could cause confusion, particularly in cases where warmups and test durations are short. In general, once the producer has reached its intended steady state throughput and does not vary too much between print windows of the test, that can be considered a sufficient warmup, and the post-warmup measurements would be reported as a separate steady-state print summary so the user can identify whether their high latency is coming from the warmup or from the steady-state operation of Kafka.

Compatibility, Deprecation, and Migration Plan

...