...

Doing this within a single file simply means parsing the file for calls to function dependencies and returning the top-level callers. Across files, this is accomplished by storing cross-file dependencies in a config file, which is written in json; when a dependency in the config file is selected, its dependents are automatically selected as well. This avoids having to parse through every test file, but it also means that the config file must be updated when cross-file dependencies are changed.

Cross-file dependencies

As mentioned above, cross-file dependencies are handled using a config file. below is an example of such a file.

{
    "tests/python/gpu/test_operator_gpu.py": [
        "test_operator.py",
        "test_optimizer.py",
        "test_random.py",
        "test_exc_handling.py",
        "test_sparse_ndarray.py",
        "test_sparse_operator.py",
        "test_ndarray.py"
    ],
    "tests/python/gpu/test_gluon_gpu.py": [
        "test_gluon.py",
        "test_loss.py",
        "test_gluon_rnn.py"
    ],
    "tests/python/mkl/test_quantization_mkldnn.py": [
        "test_quantization.py"
    ],
    "tests/python/quantization_gpu/test_quantization_gpu.py": [
        "test_quantization.py"
    ]
}

Each file with other file dependencies is listed along with a list of its dependencies. In this example, test_gluon_gpu.py depends on three files: test_gluon.py, test_loss.py, and test_gluon_rnn.py. Currently this is done manually since the structure of our tests rarely changes; however, an automated solution may be worth implementing.

...

-s SEED, --seed SEED use SEED as the test seed, rather than a random seed

Integration

The integration plan for this project can be found here: Flaky Test Bot Integration Plan. This tool will be run as a Jenkins job that will be triggered on pull requests. The Jenkins job executes the diff collator and dependency analyzer in one step to detect modified or added test cases. If any tests are selected, then MXNet is compiled and the flakiness checker is run to check the tests for flakiness.