...

Consider, however, an example of testing that seeks to understand steady state performance. One test runs for 5 minutes compared to a test with the same configuration that runs for 50 minutes. We would observe the 5-minute test has a p99 latency higher than the 50-minute test even though the steady state latency is the same. Thus, for a user to accurately measure Kafka steady-state performance, they must gather very long tests to get a “clean” p99, uncontaminated by startup latency. 
 
When analyzing steady-state latency in Kafka, it is desirable to not include startup-latency in your measurements.  Therefore, we propose adding a mechanism to optionally separate the statistics for data gathered on the first N records during warmup from those measurements collected during steady-state measurements.

Public Interfaces

No existing public interfaces are affectedThe producer performance test will add a new command line argument, --warmup-records, which will describe the number of total records from num-records, that should be treated as warmup and removed from the steady-state set of records.

Proposed Changes

We propose a modification of the producer performance test to add a command line argument for "warmup records". The total number of records for the test must be strictly greater than the number of warmup records or the test should throw an error. Negative values for warmup records should also throw an error. A value of "0" for warmup records will have the test behave identically to not specifying the parameter. Messages sent during warmup would be accumulated in a separate Stats object from the post-warmup records that make up the steady state portion of the test.

...