Page History

Flaky tests are the ever-present enemy of a build system. A flaky test is any test which does not consistently pass or fail. These tests reduce the confidence in our build. The presence of flaky failures also means that we rarely see "green builds" on Pull Requests or trunk builds. A small negative signal (one flaky failure) is amplified into a large negative signal (failing build). This means that committers become accustomed to ignoring the failing builds. The result is that other negative signals in the build (like a checkstyle failure, or compilation error) will some times get build failures. On several occasions, this has allowed other negative signals such as checkstyle or compile errors to be ignored.

We should strive to reduce, if not eliminate, flaky tests from our build.

...