Created vs Resolved Issues Chart

Table of Contents

Introduction

Goal

...

Accordig to discussions Test failures and MakeTeamcityGreenAgainGoal
To  and MakeTeamcityGreenAgain To allow community to grow more quickly and from the CI perspective, failing tests should be the main concern

For Run All dependencies at Ignite 20 Tests there should be no stable red suites. timed out suites and stable red tests.

Processing of failure

If your changes can lead to this failure(s), please create an issue with label MakeTeamCityGreenAgain and assign it to you.
If you have fixed the failure, please set the ticket to Patch Available state and write to dev list fix is ready.
For case, fix will require some time please mute test and set label Muted_Test to issue.
If you know which change caused failure please contact change's author directly .
If you don't know which change caused failure please send a message to dev list to find out

Issue Priorities
Following priorities are suggested for issues:

• Test suite timeout - it hides real test failures from us and wastes agent time
• Stable failing test/flaky - 5010%-100% of failures - issue is to be created as blocker to the next release, test may be muted after research.
• Rare Flaky tests 1%-50% 10% of failures, which are considered by teamcity as flaky may be not muted for now because TC interface helps us to identify these tests.

MTCGA Bot

For early identification of failures, for master and release branch monitoring there is special tool MTCGA Bot (Ignite TC Helper).

This tool monitors TeamCity https://ci.ignite.apache.org/ and send notifications to dev. list if it detects new failures.

Source code available at 

• ASF tools repository: https://gitbox.apache.org/repos/asf/ignite-teamcity-bot.git 
• Mirror https://github.com/apache/ignite-teamcity-bot

Ignite Developers are welcome to give feedback and post issues.

This tool can be also used for checking PR for introduction of new failures. See Apache Ignite Teamcity Bot.

Notification Types

This section covers notification types utility can send to dev@ list. User can specify his or her email and select desired tracked branches. All failures in this braches will be also forwarded to user.

New test failure notification

New test failure notification is generated if test was stable passing and then became stable failing. Stable passing test requires at least 5 sucessfull runs in a row. Stable failure requires 4 failures in a row.

Let's define 0 as success, 1 as failure. 

So history of runs ..000001111.. will cause notification. First transition 0->1 can be bug introduced test failure, and notification is linked to this particular build. Notification will not be resend in case tests continues to fail. This is done because all contributors are considered as interested in successfull tests passing and will do required steps to fix issue.

Duplicate notification can occur for same test if there is history ...000001111...000001111.  And second transition 0 ->1 can be potentially new problem, and test failure would be re-notified.

Flaky test failure notification

Some tests are flaky and sometimes change its state. This means test can be unluckily failed 4 times in a row. To protect from spam the Bot checks if fail row is happened on the one commit. If yes - test fail is considered as fail for 8 failures in a row. So history should be at least ...0000011111111...

Newly contributed tests faillure

If test has no previous history and failed 4 times in a row, then it is considered as newly introduced failure.

Critical suite failure

The Bot is able to handle timeouts & JVM crashes in suites in a special way. These type of failures are named Critical.  If 4 or more (timeouts/JMV crashes) occured several times in a row, this will generate suite-related notification. Also it is required that last run was completed with timeout.

Lets name result 2 as critical failure. So it is required to have 5 non critical failures and 4 critical to generate notification: ....(0/1)(0/1)(0/1)(0/1)(0/1) 2222....

Scope

Unassigned issues

project = IGNITE AND labels = MakeTeamcityGreenAgain AND assignee is EMPTY AND resolution is EMPTY 

 Currently

Assigned failing tests

project = IGNITE AND labels = MakeTeamcityGreenAgain AND assignee is not EMPTY AND resolution is EMPTY 

Muted failures

project = IGNITE and labels = MakeTeamcityGreenAgain and labels = Muted_test and resolution is EMPTY