THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
Below is a table indicating the number of runs needed to achieve a given confidence level with a given chance that a test passes. As a default, on CI we are using a value of 10,000 to check tests. This number will give us a relatively high confidence that we are catching flakiness even if it only occurs
success rate \ confidence | 99% | 99.9% | 99.99% |
99% | 458 | 4,603 | 46,049 |
99.9% | 687 | 6,904 | 69,074 |
99.99% | 916 | 9,205 | 92,099 |
...
python flakiness_checker.py [optional_arguments] <test-specifiers>specifier>
where <test-specifiers> specifier> is a space-separated list of test specifiers. These string specifying which test to run. This can come in two formats:
- <file-name>.<test-name>, as is common in the github repository (e.g. test_example.test_flaky)
- <directory/<file>:<test-name>, like the input to nosetests (e.g. tests/python/unittest/test_example.py:test_flaky). Note: This directory can be either relative or absolute. Additionally, if the full path is not given, the script will search whatever directory is given for the provided file.
...