...

Consider, however, an example of testing that seeks to understand steady state performance. One test runs for 5 minutes compared to a test with the same configuration that runs for 50 minutes. We would observe the 5-minute test has a p99 latency higher than the 50-minute test even though the steady state latency is the same. Thus, for a user to accurately measure Kafka steady-state performance, they must gather very long tests to get a “clean” p99, uncontaminated by startup latency. 
 
When analyzing steady-state latency in Kafka, it is desirable to not include startup-latency in your measurements.  Therefore, we propose adding a mechanism to optionally separate the statistics for data gathered on the first N records during warmup from those measurements collected during steady-state measurements.

Public Interfaces

No existing public interfaces are affected. Two options will be added to the producer performance test tool. The first option, --warmup-records, will be added to the producer performance test to request that the initial records sent in a test be gathered into a separate Stats object from the steady-state records to follow. The default value for warmup records would be 0 to ensure the continued normal operation of the producer perf test.  Only the analysis and reporting of the producer performance test may change but will require the user to opt-in and add the warmup-records parameter. A second optional parameter will also be added to the producer performance test, --separated-warmup, which will have a default value of "False". The --separated-warmup option will have the effect of separating the statistics of the warmup from the statistics of the steady state, so a warmup-only summary line would be printed followed by a steady-state-only summary line. Without invoking --separated-warmup, the first summary line printed would contain the whole test statistics, followed by a steady-state-only summary line. The default behavior of the producer performance test would be unchanged to maintain functionality for those depending on the old behavior.  

...

$ bin/kafka-producer-perf-test.sh --payload-file /opt/kafka/payload.txt --num-records 1000000 --throughput 50000
Reading payloads from: /opt/kafka/payload.txt
Number of messages read: 5000
<snip producer perf test time-series log, summary stats lines follow>
1000000 records sent, 50000.123456 records/sec (49.44 MB/sec), 5.50 ms avg latency, 600.00 ms max latency, 5 ms 50th, 10 ms 95th, 50 ms 99th, 200 ms 99.9th.

...

$ bin/kafka-producer-perf-test.sh --payload-file /opt/kafka/payload.txt --warmup-records 100000 --num-records 1000000 --throughput 50000
Reading payloads from: /opt/kafka/payload.txt
Number of messages read: 5000
Warmup first 100000 records. Steady-state results will print after the complete-test summary.
<snip producer perf test time-series log, summary stats lines follow>
1000000 records sent, 50000.123456 records/sec (49.44 MB/sec), 5.50 ms avg latency, 600.00 ms max latency, 5 ms 50th, 10 ms 95th, 50 ms 99th, 200 ms 99.9th.
900000 steady state records sent, 50000.012345 records/sec (49.32 MB/sec), 5.00 ms avg latency, 95.00 ms max latency, 5 ms 50th, 8 ms 95th, 21 ms 99th, 55 ms 99.9th.

...

$ bin/kafka-producer-perf-test.sh --payload-file /opt/kafka/payload.txt --warmup-records 100000 --num-records 1000000 --throughput 50000 --separated-warmup
Reading payloads from: /opt/kafka/payload.txt
Number of messages read: 5000
Warmup first 100000 records. Steady-state results will print after the warmup summary.
<snip producer perf test time-series log, summary stats lines follow>
100000 warmup records sent, 50000.123456 records/sec (49.44 MB/sec), 9.5 ms avg latency, 600.00 ms max latency, 5 ms 50th, 12 ms 95th, 150 ms 99th, 200 ms 99.9th.
900000 steady state records sent, 50000.012345 records/sec (49.32 MB/sec), 5.00 ms avg latency, 95.00 ms max latency, 5 ms 50th, 8 ms 95th, 21 ms 99th, 55 ms 99.9th.

...

Additional work should be described in a subsequent KIP to implement a warmup phase in the consumer performance test. 

Rejected Alternatives

An additional feature for producer warmup would be for the producer to automatically detect when its performance has stabilized then switch modes into steady state. The producer could compare variance between windows then choose to switch to steady state once the between-window variance had reached a threshold at a specified minimum. Statistics such as average latency, p99 latency, or maximum latency could be used to determine when variance has reached steady state, depending on user requirements. While this could dramatically simplify the use of the producer performance tool, automatic determination of the correct limits at which the producer should switch modes is a complex tuning problem, since that would effectively be defining steady state for all deployments. Due to this additional complexity, we believe that auto-detection of the producer’s warmup phase should be discussed in a subsequent KIP.