Page History

...

Always make sure you're able to reproduce the test failure using the random seed and environment info before you jump to a fix. This part is covered in Reproducing test results.

...

Some flaky tests could also be caused by other reasons, but the 3 reasons above cover nearly all the cases that the author has previously met. So attributing the root cause to those 3 reasons is the recommended first step for of finding root causes of flakiness.

...

As said above, the very first thing to do for this stage is to try your best to attribute the cause to the 3 major reasons.  The log should contain the name of the failed test, the random seed used in this test run, the calculated error using this formula: error = |expected - actual| / (rtol * |expected| + atol), the position where maximum error occurred, and the tolerance levels used for this test run, those are essential to identifying the root cause. Then you should move on to analyze the essential info from the test log. Usually if the error value is small (close to 1) and the tolerance levels are also very small, the problem is with the tolerance level settings. If the error is very high, it may indicate some problem with the implementations. On the other hand, if you're able to consistently re-produce the same error with the same seed, it's unlikely to be caused by a race condition problem. Otherwise, the root cause may be race conditions in the code. 

Let's take the log above as an example, we can see that the error is relatively small (close to 1), and the tolerance levels are quite small at the same time, which means the difference between actual and expected values is small and only exceeding the the allowed values by a very small amount. So for this one we can conclude that the root cause should be improper settings of tolerance levels, a quick fix can be done by bumping up the tolerance levels. The PR #12527 fixed the problem by bumping up the tolerance.

How to Fix Flakiness in tests

...