Context

With Travis CI, our builds are limited to Apache org's quota which is shared by many other Apache projects, which resulted in long waiting time and merging PRs becoming an unbearable process. Inspired by Apache Flink's approach, we are setting up a similar CI infrastructure by making use of Azure DevOps and Google Cloud's Compute Engine.

...

GCP Compute Engine (VM)

We are running a f1e2-micro (free) vm instance in this GCP project. Bootstrap steps

sudo apt update
sudo apt install git cron adoptopenjdk-8-hotspot maven
git clone git@github.com:apachehudi-ci/git-repo-sync.git
git clone git@github.com:apachehudi-ci/ci-bot.git
cd ci-bot
git checkout -t origin/fix-for-hudi
mvn clean install

Mirror master & release commits

Use crontab -e  to schedule running the jobsmirroring job. It's currently set to the following to mirror master and release commits to apachehudi-ci  repo

0*/10 * * * * $HOME/git-repo-sync/sync_repo.sh > /dev/null 2>&1
*/2 * * * * 

Scan PRs and trigger branch builds

Run $HOME/git-repo-sync/run_cibot.sh

...

to start the CI branch builds in the background.

Maintenance

Manage the background process with htop . Usually the steps include: kill the process from htop , clean up ~/ci-bot.log , and re-run the script.

Check Azure and GitHub token expiry date and update them accordingly. They are used in $HOMERun `$HOME/git-repo-sync/run_cibot.sh` to start the CI branch builds in the background.sh 

Possible issues

• Any mirrored repo (e.g., master, release-*) is not getting pushed and the mirrored CI won't run.

  • git fetch or push in $HOME/git-repo-sync/sync_repo.sh not working properly. Manually try running the git commands and troubleshoot accordingly.

Azure Pipelines

There are two pipelines defined in this Azure DevOps project

...

PR reviewer should use this CI report's result as one of the merging criteria.

Note: These PRs will be skipped for Azure CI build

• website PR targeting asf-site branch
• Labeled rfc or has [RFC-  in the title
• Labeled pr:wip or has [WIP] in the title

Get Help

• Azure DevOps docs
  • https://docs.microsoft.com/en-us/azure/devops/pipelines/?view=azure-devops
  • https://docs.microsoft.com/en-us/azure/devops/pipelines/troubleshooting/troubleshooting?view=azure-devops
• Raise questions in https://developercommunity.visualstudio.com/ for community support

...