This page is meant as a template for writing a KIP. To create a KIP choose Tools->Copy on this page and modify with your content .and replace the heading with the next KIP number and a description of your issue. Replace anything in italics with your own description
Current state: "Under Discussion"
Discussion thread: here [Change the link from the KIP proposal email archive to your own email thread]
JIRA: here [Change the link from KAFKA-1 to your own ticket]
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
When analyzing performance of long-running applications like Kafka, it is useful to separate the applications lifecycle into phases such as initialization, steady-state, transient-states due to changes in system load, and shut-down. Of these states, steady-state and transient-response-states are of special interest since these represent the normal operating states of the application. To measure the steady state performance of Kafka, one would typically set up a cluster, then create topics and proceed to measure the performance of various load points on their cluster.
A problem arises, however, when using the producer performance test to measure steady-state performance. In our observations, the producer performance test has high latency during the initialization phase as producers are coming online and brokers are beginning to exchange replicated messages. Currently, all measurements with the producer performance test will include measurements of the high latency that occurs during startup in any evaluation of steady-state performance, especially when test durations are not sufficiently long.
For example, if we observe high startup-related latency that lasts for 30 seconds into the producer performance test (about 6 prints of the 5-second window), we would need to collect at least 3000 seconds (about 50 minutes) of data for the startup latency to represent less than 1% of the latency (p99) and not affect our analysis of the steady-state latency. This 3000-second test is still problematic in the evaluation of steady-state, however, since we cannot guarantee that high startup latency will be completely resolved by the 30-second (<1%) mark, causing even these long tests to have unstable results.
Consider, however, an example of testing that seeks to understand steady state performance. One test runs for 5 minutes compared to a test with the same configuration that runs for 50 minutes. We would observe the 5-minute test has a p99 latency higher than the 50-minute test even though the steady state latency is the same. Thus, for a user to accurately measure Kafka steady-state performance, they must gather very long tests to get a “clean” p99, uncontaminated by startup latency.
When analyzing steady-state latency in Kafka, it is desirable to not include startup-latency in your measurements. Therefore, we propose adding a mechanism to optionally separate the statistics for data gathered on the first N records during warmup from those measurements collected during steady-state measurements.
No public interfaces are affected. An option such as "--warmup-records" should be added to the producer performance test to request that the initial records sent in a test be gathered into a separate Stats object from the steady-state records. The default value for warmup records would be 0 to ensure the continued normal operation of the producer perf test. Only the analysis and reporting of the producer performance test may change but would require the user to opt-in and add the warmup-records parameter.
We propose a modification of the producer performance test to add a command line argument for "warmup records". Messages sent during warmup would be accumulated in a separate Stats object from the "post-warmup" records that make up the bulk of the test. The warmup measurements would then be combined with the post-warmup latency measurements for a "full test" summary, and the post-warmup measurements would be reported as a separate steady-state print summary so the user can identify whether their high latency is coming from the warmup or from the steady-state operation of Kafka.
The proposed modification to the producer performance test would not interfere with the previous methods and command lines used to measure producer performance. Those mechanisms would not be deprecated, but the users would have a choice to either include Kafka warmup in their results or avoid the warmup latency and focus on steady-state performance.
Describe in few sentences how the KIP will be tested. We are mostly interested in system tests (since unit-tests are specific to implementation details). How will we know that the implementation works as expected? How will we know nothing broke?
If there are alternative ways of accomplishing the same thing, what were they? The purpose of this section is to motivate why the design is the way it is and not some other way.