...
This tool can be also used for checking PR for introduction of new failures.
Analyze latest TeamCity tests results against the latest master:
...
This section covers notification types utility can send to dev@ list. User can specify his or her email and select desired tracked branches. All failures in this braches will be also forwarded to user.
New test failure notification is generated if test was stable passing and then became stable failing. Stable passing test requires at least 5 sucessfull runs in a row. Stable failure requires 4 failures in a row.
...
Duplicate notification can occur for same test if there is history ...000001111...000001111. And second transition 0 ->1 can be potentially new problem, and test failure would be re-notified.
Some tests are flaky and sometimes change its state. This means test can be unluckily failed 4 times in a row. To protect from spam the Bot checks if fail row is happened on the one commit. If yes - test fail is considered as fail for 8 failures in a row. So history should be at least ...0000011111111...
If test has no previous history and failed 4 times in a row, then it should be also considered as newly introduced failure.
The Bot is able to handle timeouts & JVM crashes in suites in a special way. These type of failures are named Critical. If 4 or more (timeouts/JMV crashes) occured several times in a row, this will generate suite-related notification. Also it is required that last run was completed with timeout.
...