Page History

Note: the references to nose testing tool is outdated as the community switched to pytest for testing. See development guide.

This page is to provide some tips and tricks for fixing flaky tests. Flaky tests are defined as tests that fail intermittently on CI builds and they may indicate stability problems or improper handling of edge cases.

...

• Re-compile MXNet if necessary and verify the fix with the corresponding environment by running the same tests for more than 10000 passes (now can be done more easily with Automated Flaky Test Detector)
• Submit a PR according to MXNet Development Guide Git Setup and Workflow and/or other instructions on how to contribute to MXNet.
• Have proper title for your PR and remember to refer to the tracking issue for the flaky test in your PR
• Address any code review comments on your PR

...

After you have addressed all comments on your PR, it should be good for merge, once . Once it's merged, make sure you also check if the related Github issue is closed so that we have accurate tracking. Flakiness may still exist even after a fix is delivered as 10000 trials may still not be enough to cover each and every single random seed, so . So please stay alerted alert for future occurrences.

Appendix I: Location of test code

Logs usually comes with the test name like "test_xxx.test_yyy" where the "test_xxx" part is the name of the test file, and the "test_yyy" is the name of the actual test. All test files are located at under "tests/python/" folder. Since there are some test files that also import tests from some other test files for testing in specific environments, such as test_operator_gpu.py, there's the possibility of not being able to find a certain test in a test file. Under such case, please do a search within "tests/python/" to find where the actual test code is at if you want to make certain changes for debugging purpose or for fixing the flakiness.

...