Page History

...

Flaky tests are an ever-present problem in the Apache Kafka build. The presence of flaky tests erodes confidence in our test results which leads to many problems, both social and technical. When developers are used to seeing "red" results from the CI system, they are more likely to miss or ignore a significant test failure. The constant need for re-running the test suite in order to gain reasonable confidence puts a lot of burden on our test infrastructure – leading to very real costs. 

...

• Small amount of flakiness compounds and results in frequent build failures
• Verifying fixes for flaky tests is tedious
• Re-running all tests up to 3 times masks flakinesscan mask problems
• New tests are unknowns and frequently the source of flaky failures

...

The main idea presented here is the introduction of a test "quarantine". This is a means of isolating flaky tests so they do not affect our build outcomes. Tests that have been placed into quarantine will be run as part of the CI builds, but their results will be reported separately. Once a test is placed into quarantine, it will be evaluated for a predetermined amount of time. A test will be removed from the quarantine once it passes the exit criteria.

This idea may also be thought of as test isolation or test cordoning. However, the name quarantine seems appropriate because, like a medical quarantine, this test quarantine is designed to be temporary. 

...

Tests that have been identified as flaky will be placed into the quarantine. In this state, the tests will still be run with each CI build, but they will not cause the build to fail. A text set of text files will be used to define the current list of quarantined tests. Each Gradle project in the source tree can optionally include a a quarantined.txt file in its test resources (src/test/resources). This allows for easy collection of these quarantine files at runtime using the Java ClassLoader interface. 

...

The mechanism for skipping these tests is a JUnit ExecutionCondition that allows us to programmatically enable or disable tests.²

See Rejected Alternatives for other options of tracking our quarantined tests.

...

Quarantining New Tests

According to Google¹, flaky tests are added approximately at the same rate as they are fixed. Anecdotally, this seems to be the case in Apache Kafka as well. This phenomenon results in a non-zero steady-state failure rate in the test suite, which is not desirable.

...

Placing a new test into the quarantined will be made at the discretion of the PR authors and/or committers. It should not be compulsory. 

...

One challenge with fixing flaky tests is to know when it has actually been fixed. In some cases, there are multiple sources of flakiness within a test and it may not be obvious. There have been several occasions where Apache Kafka developers have committed a fix for a flaky test only to find that they have simply reduced the flakiness – not eliminated it. By collecting historical test data, we can increase our confidence that a fix was successful. 

Here is sample timeline for a quarantined test.

• Day 0: Flaky test identified, test is marked as quarantined. Builds no longer impacted by this test
• Day 1: Developer commits fix for flaky test
• Days 1-N: CI builds continue to run with the flaky test in quarantine, collecting data
• Day N: Test No failures after several days, the test is removed from quarantine.

For some months now, Apache Kafka has been producing Gradle Build Scans which are stored in an instance of Develocity managed host by the ASF Infra team. This application serves can serve as a historical record of our test execution. Develocity also has an API which can be used to export this data and query a broad range of information about the test suite.

In addition to the historical test execution data, we also need to know the history of which tests were marked as have been quarantined. This information can be obtained from the quarantined.txt text files and the Git history.

...

Using the Git history and the Develocity API, we can create automated reports on the state of quarantined tests.to help us identify tests that should be added or removed from the quarantine. 

New Flaky Tests

This report will query Develocity for flaky tests which only have recent history (i.e., newly committed tests). This will indicate that if a newly introduced test is flaky and should be quarantined. These are tests that should have been preemptively quarantined. 

Flaky Test Regressions

This report will query Develocity for flaky tests which have an established history and have recently started failing. This will indicate that some code (or test) change has impacted the stability of the test, or that a bug was introduced that only causes occasional test failures.

...

If a test has no flaky failures on trunk for 7 days with a minimum of 10 builds, we can consider it "cleared" and remove it from quarantine.

...