This page is meant as a template for writing a KIP. To create a KIP choose Tools->Copy on this page and modify with your content and replace the heading with the next KIP number and a description of your issue. Replace anything in italics with your own description.Status

Current state: Under DiscussionAccepted

Discussion thread: here [Change the link from the KIP proposal email archive to your own email thread]

JIRA: here [Change the link from KAFKA-1 to your own ticket]

https://lists.apache.org/thread/d4r4g9hqfmwpckdk9j01n7dh5y25j11n

JIRA:

Motivation

Flaky tests are an ever-present problem in the Apache Kafka build. The presence of flaky tests erodes confidence in our test results which leads to many problems, both social and technical. When developers are used to seeing "red" results from the CI system, they are more likely to miss or ignore a significant test failure. The constant need for re-running the test suite in order to gain reasonable confidence puts a lot of burden on our test infrastructure – leading to very real costs. 

...

• Isolation mechanism
• Historical data
• Automated reporting

Here is a diagram with the proposed workflow

Image Added

Test Isolation

There are three types of tests which we will exclude from the main test suite. They include:

...

Automatically placing a new test into the quarantine will allow us to observe its behavior without the risk of failing builds. After a few days of successful runs, these tests will graduate from the quarantine and be run as part of the main suite. Placing a new test into the quarantine will be made at the discretion of the PR authors and/or committers. It should not be compulsory. 

Historical Data

One challenge with fixing flaky tests is to know when it has actually been fixed. In some cases, there are multiple sources of flakiness within a test and it may not be obvious. There have been several occasions where Apache Kafka developers have committed a fix for a flaky test only to find that they have simply reduced the flakiness – not eliminated it. By collecting historical test data, we can increase our confidence that a fix was successful. 

...