...

This page is meant as a template for writing a KIP. To create a KIP choose Tools->Copy on this page and modify with your content and replace the heading with the next KIP number and a description of your issue. Replace anything in italics with your own description.Status

Current state: Under Discussion

...

We can break this down into a few key components:

• A test isolation Isolation mechanism
• Historical test data gathering
• Automated reporting

Test Isolation

Tests that have been identified as flaky will be placed into the quarantine. In this state, the tests will still be run with each CI build, but they will not cause the build to fail. A set of text files will be used to define the current list of quarantined tests. Each Gradle project in the source tree can optionally include a quarantined.txt file in its test resources (src/test/resources). This allows for easy collection of quarantine files at runtime using the Java ClassLoader interface. 

The quarantine files will use the following whitespace separated format

For example:

The mechanism for skipping these tests is a JUnit ExecutionCondition that allows us to programmatically enable or disable tests.¹

See Rejected Alternatives for other options of tracking our quarantined tests.

There are three types of tests which we will exclude from the main test suite. They include:

• Tests explicitly marked as flaky
• Tests recently, but no longer, marked as flaky
• Newly added integration tests

For explicitly marked flaky tests, we will use the JUnit tagging system. A new @Flaky tag will include a required String field which will refer to a Jira ticket. Explicit tagging of flaky tests makes it more obvious to developers which tests need attention, since the tags live in the source code alongside the test.

Once tests have been fixed, the tag will be removed by a developer (most likely, in the Pull Request that fixes the test). However, we don't want to include these tests in the main suite just yet. They will remain in the quarantine for 7 days to determine if they are indeed no longer flaky.

Quarantining New Tests

According to Google², flaky tests are added approximately at the same rate as they are fixed. Anecdotally, this seems to be the case in Apache Kafka as well. This phenomenon results in a non-zero steady-state failure rate in the test suite, which is not desirable.

One solution to this problem is to require a certain number of successful test runs on a Pull Request prior to merging. This is basically how we verify new tests today. The problem with this approach is that it is very ad-hoc and difficult to enforce. It also relies on a committer being heavily involved in the PR because CI runs can only be re-triggered by a committer (this is true for Jenkins and GitHub Actions). Another side effect of this approach is that it reduces our development velocity. Pull Requests stay open longer in order to produce sufficient test data. This leads to divergence from trunk which brings its own set of problems.

Placing Automatically placing a new test into the quarantine will allow us to observe its behavior without the risk of failing builds. After a few days of successful runs, these tests will graduate from the quarantine and be run as part of the main suite. 

Placing a new test into the quarantined quarantine will be made at the discretion of the PR authors and/or committers. It should not be compulsory. 

...

In addition to the historical test execution data, we also need to know the history of which tests have been quarantined. This information can be obtained from the quarantined.txt text files and the Git Develocity API and the Git history.

Reporting

Using the Git history and the Develocity API, we can create automated reports to help us identify tests that should be added or removed from the quarantine. 

New Flaky Tests

This report will query Develocity for flaky tests which only have recent history (i.e., newly committed tests). This will indicate if a newly introduced test is flaky and should be quarantined. let us see the flakiness of newly added tests. New tests with flakiness will need to promptly fixed or manually marked as flaky.

Flaky Test Regressions

This report will query Develocity for flaky tests which have an established history and have recently started failing. This will indicate that some code (or test) change has impacted the stability of the test, or that a bug was introduced that only causes occasional test failures.

...

If a test has no flaky failures on trunk for 7 days with a minimum of 10 builds, we can consider it "cleared" and remove it from quarantine.

Committer Toil

The process described in this KIP introduces a small amount of toil on the committers. 

...

from

...

quarantine

...

Since these can be batched, we can hopefully avoid a lot of small PRs.

As future work, we can consider automation that generates a Pull Request with these changes.

Long Tail Failures

As with most statistical phenomena, there is a long tail of test failures. That is to say, a relatively high number of tests which will have a very occasional failure. These failures are extremely hard to diagnose due to their rare nature. They can come from infrastructure problems, rare timing race conditions, cosmic rays, etc. Quarantining these tests would likely be ineffective because they would end up staying in the quarantine for too long. Instead, we can use the Develocity test retry feature to help with these. Rather than our existing permissive and broad application of the retries, we can use a stricter and more targeted approach.

...

Test Plan

N/A

Rejected Alternatives

Automatic Quarantining

Using the proposed test retry policy and data from Develocity, we could fully automate quarantine. The problem with this is that it makes it far too easy to ignore flaky tests. By only allowing new tests and recently fixed tests to automatically enter the quarantine, we are reducing developer toil but not making it too easy to ignore the problem.

One Quarantined Test File

Instead of one file in each project's resourcestags, we could use place a single exclusion filetext file of quarantined tests in the source tree. This has the benefit of showing all the quarantined tests in one place . The reasoning for having a file per project is so we can identify when a flaky test is added of removed on a project-by-project basis. For example, if "metadata/src/test/resources/quarantined.txt" has changes, we can tell that a metadata test has been added/removed. 

JUnit Tags

An alternative to using an exclusion file is to use JUnit tags. If we introduce a @Tag("flaky")  tag, we can utilize the test filtering included with Gradle's JUnit plugin. This has the benefit of seeing which tests are quarantined in their source files, but it makes it difficult to see all of the quarantined tests at once. This approach also makes automation more complicated because we would need to modify Java/Scala source files to add/remove a flaky annotation. By using a separate text file, we can more easily use standard unix tools and scripts for managing the flaky tests.and an easy history to look at for when tests were added and removed from the quarantine. However, the downside of this approach is that the notation of flakiness is not immediately visible to developers while looking at the test source code.