Status

Current state: "Under Discussion" Accepted

Vote thread: https://lists.apache.org/thread/pxc1sx0kb35kd8dsccc10c0pp51kp8cj

Discussion thread: DISCUSS KIP-1052 

JIRA: KAFKA-17645

Motivation

When analyzing performance of long-running applications like Kafka, it is useful to separate the application's lifecycle into phases such as initialization, steady-state, transient-states due to changes in system load, and shut-down. Of these states, steady-state and transient-response-states are of special interest since these represent the normal operating states of the application. To measure the steady state performance of Kafka, one would typically set up a cluster, then create topics and proceed to measure the performance of various load points on their cluster. 
 
A problem arises, however, when using the producer performance test to measure steady-state performance. In our observations, the producer performance test has high latency during the initialization phase as producers are coming online and brokers are beginning to exchange replicated messages. Currently, all measurements with the producer performance test will include measurements of the high latency that occurs during startup in any evaluation of steady-state performance, especially when test durations are not sufficiently long. 

...

Consider, however, an example of testing that seeks to understand steady state performance. One test runs for 5 minutes compared to a test with the same configuration that runs for 50 minutes. We would observe the 5-minute test has a p99 latency higher than the 50-minute test even though the steady state latency is the same. Thus, for a user to accurately measure Kafka steady-state performance, they must gather very long tests to get a “clean” p99, uncontaminated by startup latency. 
 
When analyzing steady-state latency in Kafka, it is desirable to not include startup-latency in your measurements.  Therefore, we propose adding a mechanism to optionally separate the statistics for data gathered on the first N records during warmup from those measurements collected during steady-state measurements.

Public Interfaces

No existing public interfaces are affected. Two options will be added to the producer performance test tool. The first option, --warmup-records, will be added to the producer performance test to request that the initial records sent in a test be gathered into a separate Stats object from the steady-state records to follow. The default value for warmup records would be 0 to ensure the continued normal operation of the producer perf test.  Only the analysis and reporting of the producer performance test may change but will require the user to opt-in and add the warmup-records parameter. A second optional parameter will also be added to the producer performance test, --separated-warmup, which will have a default value of "False". The --separated-warmup option will have the effect of separating the statistics of the warmup from the statistics of the steady state, so a warmup-only summary line would be printed followed by a steady-state-only summary line. Without invoking --separated-warmup, the first summary line printed would contain the whole test statistics, followed by a steady-state-only summary line. The default behavior of the producer performance test would be unchanged to maintain functionality for those depending on the old behavior.  

...

a new command line argument, --warmup-records, which will describe the number of total records from num-records, that should be treated as warmup and removed from the steady-state set of records.

Proposed Changes

We propose a modification of the producer performance test to add a command line argument for "warmup records". The total number of records for the test must be strictly greater than the number of warmup records or the test should throw an error. Negative values for warmup records should also throw an error.  A  A value of "0" for warmup records will have the test behave identically to not specifying the parameter. Messages sent during warmup would be accumulated in a separate Stats object from the post-warmup records that make up the steady state portion of the test.

For reporting, when the warmup-records option is invoked, an additional summary line will be printed at the end of the run. The first line printed would look identical to the existing summary print, with statistics that describe the performance of the whole test including both warmup and steady-state. This first print represents the performance of the test without a warmup and would enable relatability of new experiments to older tests that did not separate out steady state performance. The second summary line printed when warmup is invoked is the summary of the post-warmup, steady-state portion of the test. This steady-state performance could then be used to understand steady-state behavior in the absence of cluster startup costs. 

there are two options that seem to come down to user preference. Both options have two summary lines, one containing warmup data and the other being purely steady state results. In the first option, the warmup measurements could be summarized independently in a "warmup-only summary", followed by a summary of the steady state performance. With this option, the user would be able to understand performance during the warmup as a distinct phase of the measurement, then compare that to the steady state performance. The second option would be for the first summary print to combine the warmup data with the post-warmup latency measurements for a "full test summary" that is, effectively, the same output that the test currently has. This option allows users to compare what the test would have reported in the absence of a warmup and compare that to steady state performance. The first approach identifies warmup as a distinct phase, the second might be a preferred option to allow users to transition from their traditional use of the benchmark to the warmup-enabled version of the benchmark. This second option is our preferred implementation, but the behavior can be toggled with another command line option --separated-summary.  Some examples of command lines and their resulting outputs follow.  All data is fabricated but are representative of differences observed between warmup and steady state.

When run with no warmup records, the printed summary would look identical to previous versions:  

$ bin/kafka-producer-perf-test.sh --payload-file /opt/kafka/payload.txt --num-records 1000000 --throughput 50000
Reading payloads from: /opt/kafka/payload.txt
Number of messages read: 5000
<snip producer perf test time-series log, summary stats lines follow>
1000000 records sent, 50000.123456 records/sec (49.44 MB/sec), 5.50 ms avg latency, 600.00 ms max latency, 5 ms 50th, 10 ms 95th, 50 ms 99th, 200 ms 99.9th.

In the default When run with warmup-records case, without invoking --separated-warmup, the summary line printed first would be the whole test, similar to current behavior, followed by a second steady-state-only summary line:

$ bin/kafka-producer-perf-test.sh --payload-file /opt/kafka/payload.txt --warmup-records 100000 --num-records 1000000 --throughput 50000
Reading payloads from: /opt/kafka/payload.txt
Number of messages read: 5000
Warmup first 100000 records. Steady-state results will print after the complete-test summary.
<snip producer perf test time-series log, summary stats lines follow>
1000000 records sent, 50000.123456 records/sec (49.44 MB/sec), 5.50 ms avg latency, 600.00 ms max latency, 5 ms 50th, 10 ms 95th, 50 ms 99th, 200 ms 99.9th.
900000 steady state records sent, 50000.012345 records/sec (49.32 MB/sec), 5.00 ms avg latency, 95.00 ms max latency, 5 ms 50th, 8 ms 95th, 21 ms 99th, 55 ms 99.9th.

If the "--separated-warmup" option was used, the test would print a summary line first for the warmup phase, then a second summary line for the steady-state phase as follows: 

$ bin/kafka-producer-perf-test.sh --payload-file /opt/kafka/payload.txt --warmup-records 100000 --num-records 1000000 --throughput 50000 --separated-warmup
Reading payloads from: /opt/kafka/payload.txt
Number of messages read: 5000
Warmup first 100000 records. Steady-state results will print after the warmup summary.
<snip producer perf test time-series log, summary stats lines follow>
100000 warmup records sent, 50000.123456 records/sec (49.44 MB/sec), 9.5 ms avg latency, 600.00 ms max latency, 5 ms 50th, 12 ms 95th, 150 ms 99th, 200 ms 99.9th.
900000 steady state records sent, 50000.012345 records/sec (49.32 MB/sec), 5.00 ms avg latency, 95.00 ms max latency, 5 ms 50th, 8 ms 95th, 21 ms 99th, 55 ms 99.9th.

...

The proposed modification to the producer performance test would not interfere with the previous methods and command lines used to measure producer performance. Those mechanisms would not be deprecated, but the users would have a choice to either include Kafka warmup in their results or avoid the warmup latency and focus on steady-state performance.

One option will be added to the producer performance test tool. The option, --warmup-records, will be added to the producer performance test to request that the initial records sent in a test be gathered into a separate Stats object from the steady-state records to follow. The default value for warmup records would be 0 to ensure the continued normal operation of the producer perf test. Only the analysis and reporting of the producer performance test may change but will require the user to opt-in and add the warmup-records parameter. The default behavior of the producer performance test would be unchanged to maintain functionality for those depending on the old behavior.

$ bin/kafka-producer-perf-test.sh -h | grep -A1 "-warmup"
--warmup-records WARMUP-RECORDS
    The number of records to treat as warmup at the beginning of the test. If the desired test condition is one of steady state, designating warmup records will enable a second print line to describe statistics of the steady state portion. (default: 0)

Additional work should be described in a subsequent KIP to implement a warmup phase in the consumer performance test. 

...

An additional feature for producer warmup would be for the producer to automatically detect when its performance has stabilized then switch modes into steady state. The producer could compare variance between windows then choose to switch to steady state once the between-window variance had reached a threshold at a specified minimum. Statistics such as average latency, p99 latency, or maximum latency could be used to determine when variance has reached steady state, depending on user requirements Mechanisms for automatic detection of producer stability at end of warmup include simple checks such as time-based warmup or latency-based warmup, but monitored performance indicators could also include DNS queries, broker SSL connections, Producer Throttler actions, response intervals from brokers, and even JVM compilation activity. While this could dramatically simplify the use of the producer performance tool, automatic determination of the correct limits at which the producer should switch modes is a complex tuning problem, since that would effectively be defining steady state for all deployments. Due to this additional complexity, we believe that auto-detection of the producer’s warmup phase should be discussed in a subsequent KIP.

Another rejected alternative would be for an additional option that controls whether to separate the warmup portion of the test from the steady-state portion of the test in the printed lines. This option is rejected because, when studying steady-state the performance of the warmup portion of the test is, essentially, nonsensical and does not hold analytical value. The warmup option already causes a steady-state summary to be printed following the whole-test summary print. In contrast, when studying warmup, users can already isolate warmup behavior by running short tests that focus only on the first minute of test warmup.