...

When analyzing performance of long-running applications like Kafka, it is useful to separate the applications application's lifecycle into phases such as initialization, steady-state, transient-states due to changes in system load, and shut-down. Of these states, steady-state and transient-response-states are of special interest since these represent the normal operating states of the application. To measure the steady state performance of Kafka, one would typically set up a cluster, then create topics and proceed to measure the performance of various load points on their cluster. 
 
A problem arises, however, when using the producer performance test to measure steady-state performance. In our observations, the producer performance test has high latency during the initialization phase as producers are coming online and brokers are beginning to exchange replicated messages. Currently, all measurements with the producer performance test will include measurements of the high latency that occurs during startup in any evaluation of steady-state performance, especially when test durations are not sufficiently long. 

...

No public interfaces are affected. An option such as ". Two options will be added to the producer performance test tool. The first option, --warmup-records" should , will be added to the producer performance test to request that the initial records sent in a test be gathered into a separate Stats object from the steady-state records to follow. The default value for warmup records would be 0 to ensure the continued normal operation of the producer perf test.  Only the analysis and reporting of the producer performance test may change but would will require the user to opt-in and add the warmup-records parameter. A second optional parameter will also be added to the producer performance test, --separated-warmup, which will have a default value of "False". The --separated-warmup option will have the effect of separating the statistics of the warmup from the statistics of the steady state, so a warmup-only summary line would be printed followed by a steady-state-only summary line. Without invoking --separated-warmup, the first summary line printed would contain the whole test statistics, followed by a steady-state-only summary line. The default behavior of the producer performance test would be unchanged to maintain functionality for those depending on the old behavior.  

$ bin/kafka-producer-perf-test.sh -h | grep -A1 "-warmup"
--warmup-records WARMUP-RECORDS
    The number of records to treat as warmup at the beginning of the test. If the desired test condition is one of steady state, designating warmup records will enable a second print line to describe statistics of the steady state portion. (default: 0)
--separated-warmup
    When using warmup records, invoking this option will separate the warmup phase into its own statistics and print a summary for it. Without using --separated-warmup, the first summary line to print will be the summary of the entire test, followed by the summary of the steady-state portion of the test. The default option (false) is to have the first printed summary line report statistics on the combined warmup and steady-state phases, followed by the steady-state-only summary line.

Proposed Changes

We propose a modification of the producer performance test to add a command line argument for "warmup records". The total number of records for the test must be strictly greater than the number of warmup records or the test should throw an error. Negative values for warmup records should also throw an error.  A value of "0" for warmup records should will have the test behave identically to not specifying the parameter. Messages sent during warmup would be accumulated in a separate Stats object from the "post-warmup " records that make up the steady state portion of the test.

For reporting, there are two options that seem to come down to user preference. Both options have two summary lines, one containing warmup data and the other being purely steady state results.   In the first option, the warmup measurements could be summarized independently in a "warmup-only summary", followed by a summary of the steady state performance. With this option, the user would be able to understand performance during the warmup as a distinct phase of the measurement, then compare that to the steady state performance. The second option would be for the first summary print to combine the warmup data with the post-warmup latency measurements for a "full test summary" that is, effectively, the same output that the test currently has. This option allows users to compare what the test would have reported in the absence of a warmup and compare that to steady state performance. The first approach identifies warmup as a distinct phase, the second might be a preferred option to allow users to transition from their traditional use of the benchmark to the warmup-enabled version of the benchmark. This second option is our preferred implementation, but the behavior could can be toggled with another command line option such as '--combined-summary' --separated-summary.  Some examples of command lines and their resulting outputs follow.  All data is fabricated but are representative of differences observed between warmup and steady state.

When run with no warmup records, the printed summary would look identical to previous versions: 

$ bin/kafka-producer-perf-test.sh --num-records 1000000 --throughput 50000
Reading payloads from: /opt/kafka/payload.txt
Number of messages read: 5000
<snip producer perf test time-series log, summary stats lines follow>
1000000 records sent, 50000.123456 records/sec (49.44 MB/sec), 5.50 ms avg latency, 600.00 ms max latency, 5 ms 50th, 10 ms 95th, 50 ms 99th, 200 ms 99.9th.

In the default warmup-records case, without invoking --separated-warmup, the summary line printed first would be the whole test, similar to current behavior, followed by a second steady-state-only summary line:

$ bin/kafka-producer-perf-test.sh --warmup-records 100000 --num-records 1000000 --throughput 50000
Reading payloads from: /opt/kafka/payload.txt
Number of messages read: 5000
Warmup first 100000 records. Steady-state results will print after the complete-test summary.
<snip producer perf test time-series log, summary stats lines follow>
1000000 records sent, 50000.123456 records/sec (49.44 MB/sec), 5.50 ms avg latency, 600.00 ms max latency, 5 ms 50th, 10 ms 95th, 50 ms 99th, 200 ms 99.9th.
900000 steady state records sent, 50000.012345 records/sec (49.32 MB/sec), 5.00 ms avg latency, 95.00 ms max latency, 5 ms 50th, 8 ms 95th, 21 ms 99th, 55 ms 99.9th.

If the "--separated-warmup" option was used, the test would print a summary line first for the warmup phase, then a second summary line for the steady-state phase as follows: 

$ bin/kafka-producer-perf-test.sh --warmup-records 100000 --num-records 1000000 --throughput 50000 --separated-warmup
Reading payloads from: /opt/kafka/payload.txt
Number of messages read: 5000
Warmup first 100000 records. Steady-state results will print after the warmup summary.
<snip producer perf test time-series log, summary stats lines follow>
100000 warmup records sent, 50000.123456 records/sec (49.44 MB/sec), 9.5 ms avg latency, 600.00 ms max latency, 5 ms 50th, 12 ms 95th, 150 ms 99th, 200 ms 99.9th.
900000 steady state records sent, 50000.012345 records/sec (49.32 MB/sec), 5.00 ms avg latency, 95.00 ms max latency, 5 ms 50th, 8 ms 95th, 21 ms 99th, 55 ms 99.9th.

Regarding the process of choosing a warmup duration, the warmup records should comprise the time required for producers to startup, allocate resources, connect to the brokers, then reach steady state performance. Although a one-minute warmup is common in Java performance analysis, a simple process for more accurately determining the warmup time of a given topology would be to run the test without a warmup first. The resulting time-series of the producer performance test output would typically have high latency during the first few windows of output (at 5 s per window). The minimum warmup records could then be chosen to be the number of records sent once producer latency reaches an acceptable steady state. Although the producer performance test output should provide sufficient information to set a warmup, additional data sources such as JMX:producer:record-queue-time, JMX:producer:request-latency-avg, and the various broker queue-time metrics could be consulted to help precisely determine when steady state has been achieved. Since the warmup data will be reported separately from the desired steady state results, and running longer warmups should not impact steady state performance, it's always good practice to run a warmup for slightly longer than is deemed necessary to guarantee records from the warmup have been processed and ensure steady state operation.

...

The proposed modification to the producer performance test would not interfere with the previous methods and command lines used to measure producer performance. Those mechanisms would not be deprecated, but the users would have a choice to either include Kafka warmup in their results or avoid the warmup latency and focus on steady-state performance.

Additional work should be described in a subsequent KIP to implement a warmup phase in the consumer performance test. 

Rejected Alternatives

.