Page History

...

Automatically placing a new test into the quarantine will allow us to observe its behavior without the risk of failing builds. After a few days of successful runs, these tests will graduate from the quarantine and be run as part of the main suite. 

Placing a new test into the quarantine will be made at the discretion of the PR authors and/or committers. It should not be compulsory. 

Historical Data

One challenge with fixing flaky tests is to know when it has actually been fixed. In some cases, there are multiple sources of flakiness within a test and it may not be obvious. There have been several occasions where Apache Kafka developers have committed a fix for a flaky test only to find that they have simply reduced the flakiness – not eliminated it. By collecting historical test data, we can increase our confidence that a fix was successful. 

...