...

The mechanism for skipping these tests is a JUnit ExecutionCondition that allows us to programmatically enable or disable tests.²¹

See Rejected Alternatives for other options of tracking our quarantined tests.

Quarantining New Tests

According to Google¹², flaky tests are added approximately at the same rate as they are fixed. Anecdotally, this seems to be the case in Apache Kafka as well. This phenomenon results in a non-zero steady-state failure rate in the test suite, which is not desirable.

...

As with most statistical phenomena, there is a long tail of test failures. That is to say, a relatively high number of tests which will have a very occasional failure. These failures are extremely hard to diagnose due to their rare nature. They can come from infrastructure problems, rare timing race conditions, cosmic rays, etc. Quarantining these tests would likely be ineffective because they would end up staying in the quarantine for too long. Instead, we can use the Develocity test retry feature to help with these. Rather than our existing permissive and broad application of the retries, we can use a stricter and more targeted approach.

References

1. https://junit.org/junit5/docs/current/user-guide/#extensions-conditions
2. https://testing.googleblog.com/2016/05/flaky-tests-at-google-and-how-we.html
3. https://medium.com/junit.org/junit5/docs/current/user-guide/#extensions-conditions@tom.tikkle/how-to-manage-flaky-tests-quarantine-a34c86993354
4. https://medium.com/scopedev/how-can-we-peacefully-co-exist-with-flaky-tests-3c8f94fba166

Compatibility, Deprecation, and Migration Plan

...