This page is meant as a template for writing a KIP. To create a KIP choose Tools->Copy on this page and modify with your content and replace the heading with the next KIP number and a description of your issue. Replace anything in italics with your own description.
Current state: Under Discussion
Discussion thread: here [Change the link from the KIP proposal email archive to your own email thread]
JIRA: here [Change the link from KAFKA-1 to your own ticket]
Please keep the discussion on the mailing list rather than commenting on the wiki (wiki discussions get unwieldy fast).
Flaky tests are an ever-present problem in the Apache Kafka build. The presence of flaky tests erodes confidence in our test results which leads to many problems, both social and technical. When developers are used to seeing "red" results from the CI system, they are more likely to miss or ignore a significant test failure. The constant need for re-running the test suite in order to gain reasonable confidence puts a lot of burden on our test infrastructure – leading to very real costs.
The aim of this KIP is to address some of the technical problems of flaky tests, which will hopefully alleviate some the social problems.
Some of our technical problems include:
No public interfaces are affected by this KIP. This is a development process change.
The main idea presented here is the introduction of a test "quarantine". This is a means of isolating flaky tests so they do not affect our build outcomes. Tests that have been placed into quarantine will be run as part of the CI builds, but their results will be reported separately. Once a test is placed into quarantine, it will be evaluated for a predetermined amount of time. A test will be removed from the quarantine once it passes the exit criteria.
This idea may also be thought of as test isolation or test cordoning. However, the name quarantine seems appropriate because, like a medical quarantine, this test quarantine is designed to be temporary.
We can break this down into a few key components:
Tests that have been identified as flaky will be placed into the quarantine. In this state, the tests will still be run with each CI build, but they will not cause the build to fail. A set of text files will be used to define the current list of quarantined tests. Each Gradle project in the source tree can optionally include a quarantined.txt file in its test resources (src/test/resources). This allows for easy collection of quarantine files at runtime using the Java ClassLoader interface.
The quarantine files will use the following whitespace separated format
DATE " " FULLY_QUALIFIED_CLASS_NAME [ "#" METHOD_NAME ] [ " " REASON ] |
For example:
2024-09-18 org.apache.kafka.controller.QuorumControllerTest#testFenceMultipleBrokers KAFKA-16634 |
The mechanism for skipping these tests is a JUnit ExecutionCondition that allows us to programmatically enable or disable tests.1
See Rejected Alternatives for other options of tracking our quarantined tests.
According to Google2, flaky tests are added approximately at the same rate as they are fixed. Anecdotally, this seems to be the case in Apache Kafka as well. This phenomenon results in a non-zero steady-state failure rate in the test suite, which is not desirable.
One solution to this problem is to require a certain number of successful test runs on a Pull Request prior to merging. This is basically how we verify new tests today. The problem with this approach is that it is very ad-hoc and difficult to enforce. It also relies on a committer being heavily involved in the PR because CI runs can only be re-triggered by a committer (this is true for Jenkins and GitHub Actions). Another side effect of this approach is that it reduces our development velocity. Pull Requests stay open longer in order to produce sufficient test data. This leads to divergence from trunk which brings its own set of problems.
Placing a new test into the quarantine will allow us to observe its behavior without the risk of failing builds. After a few days of successful runs, these tests will graduate from the quarantine and be run as part of the main suite.
Placing a new test into the quarantined will be made at the discretion of the PR authors and/or committers. It should not be compulsory.
One challenge with fixing flaky tests is to know when it has actually been fixed. In some cases, there are multiple sources of flakiness within a test and it may not be obvious. There have been several occasions where Apache Kafka developers have committed a fix for a flaky test only to find that they have simply reduced the flakiness – not eliminated it. By collecting historical test data, we can increase our confidence that a fix was successful.
Here is sample timeline for a quarantined test.
For some months now, Apache Kafka has been producing Gradle Build Scans which are stored in an instance of Develocity host by the ASF. This application can serve as a historical record of our test execution. Develocity also has an API which can be used to export this data and query a broad range of information about the test suite.
In addition to the historical test execution data, we also need to know the history of which tests have been quarantined. This information can be obtained from the quarantined.txt text files and the Git history.
Using the Git history and the Develocity API, we can create automated reports to help us identify tests that should be added or removed from the quarantine.
New Flaky Tests
This report will query Develocity for flaky tests which only have recent history (i.e., newly committed tests). This will indicate if a newly introduced test is flaky and should be quarantined.
Flaky Test Regressions
This report will query Develocity for flaky tests which have an established history and have recently started failing. This will indicate that some code (or test) change has impacted the stability of the test, or that a bug was introduced that only causes occasional test failures.
Cleared Tests
This report will query Develocity for passing tests that are on the quarantined list. If a test has had passing builds above some threshold, it will be reported as "cleared" from the quarantine.
Defective Tests
This report will list tests that have been quarantined for an excessive number of days. Tests in this report should be considered for removal or rewriting. This report will help surface tests that have been in the quarantine for too long and perhaps been ignored or forgotten about.
If a test has no flaky failures on trunk for 7 days with a minimum of 10 builds, we can consider it "cleared" and remove it from quarantine.
The process described in this KIP introduces a small amount of toil on the committers.
Since these can be batched, we can hopefully avoid a lot of small PRs.
As future work, we can consider automation that generates a Pull Request with these changes.
As with most statistical phenomena, there is a long tail of test failures. That is to say, a relatively high number of tests which will have a very occasional failure. These failures are extremely hard to diagnose due to their rare nature. They can come from infrastructure problems, rare timing race conditions, cosmic rays, etc. Quarantining these tests would likely be ineffective because they would end up staying in the quarantine for too long. Instead, we can use the Develocity test retry feature to help with these. Rather than our existing permissive and broad application of the retries, we can use a stricter and more targeted approach.
N/A
N/A
One Quarantined Test File
Instead of one file in each project's resources, we could use a single exclusion file. This has the benefit of showing all the quarantined tests in one place. The reasoning for having a file per project is so we can identify when a flaky test is added of removed on a project-by-project basis. For example, if "metadata/src/test/resources/quarantined.txt" has changes, we can tell that a metadata test has been added/removed.
JUnit Tags
An alternative to using an exclusion file is to use JUnit tags. If we introduce a @Tag("flaky") tag, we can utilize the test filtering included with Gradle's JUnit plugin. This has the benefit of seeing which tests are quarantined in their source files, but it makes it difficult to see all of the quarantined tests at once. This approach also makes automation more complicated because we would need to modify Java/Scala source files to add/remove a flaky annotation. By using a separate text file, we can more easily use standard unix tools and scripts for managing the flaky tests.