...

In order to test a test for it's robustness against flakynessflakiness, you might want to repeat the execution multiple times. This can be achieved with the MXNET_TEST_COUNT environment variable. The execution would look like follows:

...

MXNET_TEST_SEED=2096230603 nosetests --logging-level=DEBUG --verbose -s test_module.py:test_op3

Using the Flakiness Checker

Another way to accomplish the above is with the flakiness checker tool, which is currently located in the tools directory. This automatically sets the correct environment variables and infers the path.

Similar results to the above can be achieved using the following commands:

python tools/flakiness_checker.py test_module.test_op3
python tools/flakiness_checker.py test_module.test_op3 -s 2096230603

Usage documentation:

python tools/flakiness_checker.py [optional_arguments] <test-specifier>

where <test-specifier> is a string specifying which test to run. This can come in two formats:

1. <file-name>.<test-name>, as is common in the github repository (e.g. test_example.test_flaky)
2. <directory/<file>:<test-name>, like the input to nosetests (e.g. tests/python/unittest/test_example.py:test_flaky). Note: This directory can be either relative or absolute. Additionally, if the full path is not given, the script will search whatever directory is given for the provided file.

Optional Arguments:

-h, --help print built-in help message

-n N, --num-trials N run test for n trials, instead of the default of 10,000

-s SEED, --seed SEED use SEED as the test seed, rather than a random seed

 Note: additional options will be added once the flaky test detector is deployed

Troubleshooting

In case you run into any issues, please try the following steps:

...