TeamCity

JetBrains TeamCity is the main CI/CD service for building, testing and releasing Apache Ignite and it's components.

TeamCity 1

URL: https://ci.ignite.apache.org/

TeamCity 2

URL: https://ci2.ignite.apache.org/

TC1 → TC2 syncing

Current scheme of keeping consistency between 2 instances of TeamCity instances.

...

1. After 1^st sync it is required to enter secrets value represented by Tokens (in Versions Settings option).
   Current list of secrets:
   • secure:password in synchronization VCS root repository settings: GitHub token for user which have access to sync repository
   • secure:nuget.api.key in [Apache Ignite 2.x] / [Release] / [Main] / [3] Build & Upload Nuget Staging Packages: nuget.org API key for uploading released NuGet packages
   • MYGET_PASSWORD in [Apache Ignite 2.x] / [Release] / [Nightly] / [APACHE IGNITE NIGHTLY RELEASE] #2 :: Assemble Binaries: myget.org password for uploading nightly Maven packages
   • secure:github_access_token in [Apache Ignite 3.x] / [Test] / --> Run :: All Tests: GitHub token (Valentin Kulichenko's) for integrating TeamCity and GitHub in terms of reporting tests results to PR
2. Instead of entering values in Tokens, it is possible to copy values from sources TeamCity instances as follows:

   1. Search for credentialsJSON settings in repository with settings
      

      Code Block
      language bash
      grep -R 'credentialsJSON:' .teamcity/ | \ sed -r 's|(.):.param name="(.)".value="(.)" .|\1 :: \2=\3|' | \ sed -r 's| spec=.||'

      
      

   2. Search for occurrences of UUIDs in <source-teamcity-host>:/<teamcity-data-dir>/config/project
      

      Code Block
      language bash
      cd /<source-teamcity-data-dir>/config/project find . -name credentials.json | while read file; do echo "${file}" grep --color -E '(<UUID-1>|UUID-2|...|<UUID-n>)|$' "${file}" done

      
      

   3. Remove not matched UUIDs (they will not be colored) by editing files manually.

   4. Copy updated credentials.json files to some folder

      Code Block
      language bash
      cd /<source-teamcity-data-dir>/config/project mkdir -pv /tmp/credentialsJSON find . -name credentials.json | while read file; do    cp -v --parents "${file}" /tmp/credentialsJSON done

      
      

   5. Pack secrets in archive

      Code Block
      language bash
      cd /tmp/credentialsJSON zip -r ../credentialsJSON.zip *

      
      

   6. Copy archive to target TeamCity instance <source-teamcity-host>:/<teamcity-data-dir>/config/project

   7. Find local secrets and remove them

      Code Block
      language bash
      cd /<target-teamcity-data-dir>/config/project find . -name credentials.json -exec rm -v {} \; unzip credentialsJSON.zip *

      
      

   8. Restart TeamCity instance

      Code Block
      language bash
      cd /<target-teamcity-server-dir> bash bin/teamcity-server.sh stop ... bash bin/teamcity-server.sh start

      
      

   9. Check /<target-teamcity-server-dir>/logs/teamcity-server.log that TeamCity instance started correctly

Users / Groups / Roles

Synchronization of users, groups and roles requires separate export and import procedure.

1. Dump specific tables from source TeamCity database

   Code Block
   language bash
   mysqldump --no-tablespaces -u teamcity -p teamcity ids_group \ ids_group_entity_id \ project \ project_mapping \ user_blocks \ user_build_parameters \ user_build_types_order \ usergroup_notification_data \ usergroup_notification_events \ usergroup_roles \ usergroups \ usergroup_subgroups \ usergroup_users \ usergroup_watch_type \ user_notification_data \ user_notification_events \ user_projects_visibility \ user_property \ user_roles \ users \ user_watch_type \ vcs_username > teamcity.sql

   
   

2. Copy /<source-teamcity-data-dir>/config/roles-config.xml

3. Pack dump and configuration fail into archive

   Code Block
   language bash
   zip -r users-groups-roles.zip teamcity.sql roles-config.xml

   
   

4. Copy archive to target TeamCity instance <source-teamcity-host>

5. Unpack archive
   

   Code Block
   language bash
   unzip users-groups-roles.zip

   
   

6. Place roles-config.xml to /<target-teamcity-data-dir>/config/roles-config.xml

7. Restore specific tables (TeamCity and/or DB service stop is not required)

   Code Block
   language bash
   mysql -u teamcity -p teamcity < teamcity.sql

   
   

8. Restart TC (restart is required to re-read some DB data which is not updated on-the-fly).

   1. Stop queue

      Code Block
      language bash
      curl -s \ -u "<tc_username>:<tc_password>" \ -X POST \ "https://<tc_url>/queueStartStop.html?stateChangeReason=TeamCity+daily+maintenance&newQueueState=false"

      
      

   2. Wait for non-composite running builds to finish

      Code Block
      language bash
      while true; do runningBuildsCount="$(curl -s \ -u "<tc_username>:<tc_password>" \ -X GET \ "https://<tc_url>/app/rest/builds/?locator=state:running,composite:false" | \ xmllint --format \ --xpath "string(/*[local-name()='builds']/@count)" -)" if [ "$runningBuildsCount" -gt 0 ]; then echo "Running builds left: ${runningBuildsCount}" sleep 60 else break fi done

      
      

   3. Restart server

      Code Block
      language bash
      curl -s \ -u "<tc_username>:<tc_password>" \ -X POST \ "https://<tc_url>/admin/serverRestart.html?start=1" 1>/dev/null 2>&1 sleep 10

      
      

   4. Wait for TeamCity to boot

      Code Block
      language bash
      while true; do returnCode="$(curl -s \ -I \ -u "<tc_username>:<tc_password>" \ -o /dev/null \ -w "%{http_code}" \ -X GET \ "https://<tc_url>/app/rest/server")" if [ "${returnCode}" != "200" ]; then echo "Current TeamCity HTTP status: ${returnCode}" sleep 60 else break fi done

      
      

   5. Resume queue

      Code Block
      language bash
      curl -s \ -u "<tc_username>:<tc_password>" \ -X POST \ "https://<tc_url>/queueStartStop.html?stateChangeReason=&newQueueState=true"

      
      

Backup

TeamCity server has embedded backup tool which can backup database, project files and settings and build logs and artifacts.

Backup can be done in two ways:

1. Via UI: https://ci.ignite.apache.org/admin/admin.html?item=backup

2. Via REST API:

   1. (optional) Remove old backups

      Code Block
      language bash
      find "<teamcity-data_dir>/backup/" -type f -ctime +<days> -exec rm -rfv {} \; | sed -r 's|^| - |'

      
      

   2. Initiate backup

      Code Block
      language bash
      curl -s \ -u "<tc_username>:<tc_password>" \ -X POST \ "https://<tc_url>/app/rest/server/backup?includeConfigs=true&includeBuildLogs=false&fileName=<tc_backup_name>"

      
      

...

Travis CI

...

Backlog issues

1. TC: Introduce automated instructions for preparing Linux and Windows agents.
2. ✅ TC: Remove obsolete secret tokens.
3. TC: Introduce Kotlin DSL based settings (instead of XML).