Status

Current state: "Voting in Progress" Accepted

Vote thread: https://lists.apache.org/thread/pxc1sx0kb35kd8dsccc10c0pp51kp8cj

Discussion thread: DISCUSS KIP-1052 

JIRA: KAFKA-17645

Motivation

When analyzing performance of long-running applications like Kafka, it is useful to separate the application's lifecycle into phases such as initialization, steady-state, transient-states due to changes in system load, and shut-down. Of these states, steady-state and transient-response-states are of special interest since these represent the normal operating states of the application. To measure the steady state performance of Kafka, one would typically set up a cluster, then create topics and proceed to measure the performance of various load points on their cluster. 
 
A problem arises, however, when using the producer performance test to measure steady-state performance. In our observations, the producer performance test has high latency during the initialization phase as producers are coming online and brokers are beginning to exchange replicated messages. Currently, all measurements with the producer performance test will include measurements of the high latency that occurs during startup in any evaluation of steady-state performance, especially when test durations are not sufficiently long. 

...

Consider, however, an example of testing that seeks to understand steady state performance. One test runs for 5 minutes compared to a test with the same configuration that runs for 50 minutes. We would observe the 5-minute test has a p99 latency higher than the 50-minute test even though the steady state latency is the same. Thus, for a user to accurately measure Kafka steady-state performance, they must gather very long tests to get a “clean” p99, uncontaminated by startup latency. 
 
When analyzing steady-state latency in Kafka, it is desirable to not include startup-latency in your measurements.  Therefore, we propose adding a mechanism to optionally separate the statistics for data gathered on the first N records during warmup from those measurements collected during steady-state measurements.

Public Interfaces

No existing public interfaces are affectedThe producer performance test will add a new command line argument, --warmup-records, which will describe the number of total records from num-records, that should be treated as warmup and removed from the steady-state set of records.

Proposed Changes

We propose a modification of the producer performance test to add a command line argument for "warmup records". The total number of records for the test must be strictly greater than the number of warmup records or the test should throw an error. Negative values for warmup records should also throw an error. A value of "0" for warmup records will have the test behave identically to not specifying the parameter. Messages sent during warmup would be accumulated in a separate Stats object from the post-warmup records that make up the steady state portion of the test.

...

An additional feature for producer warmup would be for the producer to automatically detect when its performance has stabilized then switch modes into steady state. The producer could compare variance between windows then choose to switch to steady state once the between-window variance had reached a threshold at a specified minimum. Statistics such as average latency, p99 latency, or maximum latency could be used to determine when variance has reached steady state, depending on user requirements Mechanisms for automatic detection of producer stability at end of warmup include simple checks such as time-based warmup or latency-based warmup, but monitored performance indicators could also include DNS queries, broker SSL connections, Producer Throttler actions, response intervals from brokers, and even JVM compilation activity. While this could dramatically simplify the use of the producer performance tool, automatic determination of the correct limits at which the producer should switch modes is a complex tuning problem, since that would effectively be defining steady state for all deployments. Due to this additional complexity, we believe that auto-detection of the producer’s warmup phase should be discussed in a subsequent KIP.

...