Apache Flink's is powered by Azure pipelines and currently hosted by Ververica/Alibaba.

Repositories

AzureCI requires write access to the repository, but the Apache Foundation has certain restrictions who's able to access the public repository. Therefore, a workaround is provided that requires multiple Github repositories:

• apache/flink: The actual Apache Flink repository. It's hosted under the Apache Github organization.
• flink-ci is a Ververica-owned Github organization that hosts multiple repositories relevant for Apache Flink's CI infrastructure
  • flink-ci/flink-mirror: Mirror repository of apache/flink . The AzureCI account has access to this repository and can run CI builds on the corresponding code base.
  • flink-ci/flink: Mirror repository of apache/flink  for PR branch synchronization. PR CI runs don't have the CI secrets configured. Tests requiring secrets will only run on master and the release branches.
  • flink-ci/git-repo-sync: Initial effort around synchronizing apache/flink  and flink-ci/flink-mirror. This script is still in use to synchronize master and the release branches of the two most-recently published Flink versions. The actually script execution is hosted on Ververica-hosted machine.
  • flink-ci/cibot: Repository for the CIBot which is another synchronization mechanism that's in charge of synchronizing branches of PRs that were opened on apache/flink. The bot is also in charge of updating the CI information of those PRs. The bot is hosted on a Ververica-hosted machine.
  • flink-ci/pr-labeler: Repository for the PR labeler. The PR labeler is used to add Flink modules as labels to the PRs. A version of the pr-labeler  is deployed onto a Ververica-hosted machine.

AzureCI Infrastructure

We use Azure pipelines for our Continuous Integration.

The jobs are defined in the azure-pipelines.yml file located in the project's root directory. Several templates also exist in the tools/azure-pipelines directory.

Builds for the apache/flink  repository run on Ververica-sponsored Azure account apache-flink.

Builds for branches run in the flink-ci mirror of the flink repository.

Builds for pull requests are mirrored into the flink-ci repository using CiBot. This is currently necessary because the ASF does not allow Azure to be used directly because it requires write permissions to the repo.

The running builds can be accessed through the AzureCI web UI.

The jobs are defined in the apache/flink:azure-pipelines.yml. Several templates also exist in the apache/flink:tools/azure-pipelines/ directory.

Azure CI Agents

Apache Flink's AzureCI has two different types of workers which are used for the CI runs (the agent that is used for running the job is usually logged in the first line of a job run's log):

• AzureCI hosts: AzureCI provides VMs to OpenSource projects
• Alibaba-hosted machines: Alibaba provides additional VMs to host Flink's CI builds.

Future Plans

There are plans to migrate from AzureCI to Github Actions to provide a better integrated look-and-feel for developers (

Contacts

• Robert Metzger (PMC chair)
• Chesnay Schepler (PMC): one of the main maintainers of the CI infrastructure
• Jing Ge: Ververica contact person
• Yang Wang (PMC): Alibaba contact person who has access to dedicated AzureCI agents

...

• .

Defined environment variables are usually controlled by the PMC; if you require access to the original keys please reach out to the PMC. Please do not add additional keys without prior notice, nor pass them on to other people as they are for the most part credentials to sponsored resources.