Highlights
The location ./test-infra/jenkins
contains the defined Jenkins jobs are defined in These jobs are written using the Job DSL using Apache Groovy. Job definitions should be a simple as possible and ideally identify a single Gradle target to execute.
Testing Changes
The following are some Testing Changes tips that could help you:
- Use Local Dockerized Jenkins to test changes locally without breaking the project’s job definitions.
- Test a PR that includes Jenkins changes, but it is potentially destructive because the job definitions are a shared resource.
- To update the job definitions on a PR we need to run the Seed Job. This will cause the job definitions to be read and parsed. Running a seed job is a high privilege workflow, so it can only be run by committers.
- To run a seed job over a committer PR, one needs only to type "Run Seed Job" into the PR in Github.
- To run a seed job over a non-commiter PR, one needs to follow a hacky workaround: https://drive.google.com/file/d/1Yk4TIF3Qw_ur4P8fTx1I1469TDFdCHHq/view
- Another option is to have a committer clone the PR and type "Run Seed Job"
Triggering jobs
Beam committers can trigger a job with the Jenkins UI. Non-committers can trigger a job if there is a trigger phrase.
- To trigger a job on Jenkins UI, log in on https://ci-beam.apache.org/, find the job page with search, select Build with Parameters page from the menu and click Build (you may also specify a build parameter, for example a commit sha1 value).
- Alternatively, to reliably trigger at a specific PR, select Configure and add
ghprbPullId
as a String parameter. Then set the default for sha1 toorigin/pr/${ghprbPullId}/merge
. Save. Then select Build with Parameters, and put in the PR number.
Job suffixes
Each post-commit and pre-commit job file defines several jobs with different suffixes.
- For pre-commits, there are
_Commit
,_Phrase
, and_Cron
suffixes. - The
_Commit
job happens with every push to a pull request. - The
_Phrase
happens when the trigger phrase is entered as a comment in the pre-commit. - The
_Cron
pre-commit happens post-commit on the master branch as a signal of whether the pre-commit would pass without any changes.
Job labels
Consider the following tips regarding Jenkins-jobs-related tags:
- Most Beam Jenkins jobs specify the label
beam
, which uses beam executors 1-15. - Performance test jobs specify the label
beam-perf
, which uses beam executors 16.- Used for tests that run locally on the Jenkins worker such as Direct Runner. Configured to run a one job at a time to reduce the noisy neighbor problem
- Website publishing job specifies the label
git-websites
, which allows publishing generated documentation to the asf-site branch.
How to access a test history
To access a test history:
- Look at this URL:
https://ci-beam.apache.org/job/beam_PreCommit_Java_Phrase/lastCompletedBuild/testReport/junit/org.apache.beam.runners.fnexecution.data/GrpcDataServiceTest/testMessageReceivedBySingleClientWhenThereAreMultipleClients/history/ - Go to failed job -> test result -> navigate to failed test -> history
How to delete jobs that are no longer needed
To delete jobs that are no longer needed in Jenkins:
Remove the
`.groovy`
file associated with the job. For more information, see- Delete the job history from Jenkins (committers only, if you are not a committer please ask the reviewer of your PR to help with this).
- Open https://ci-beam.apache.org/ and log in with your Apache Credentials.
- Select the job you want to remove.
- Click Delete project in the left side panel.
Jenkins Infrastructure Setup
16 GCE instances support Jenkins' continuous integrating jobs. The instance group 'apache-ci-beam-jenkins-nodes'
manages the compute instances in the Google Cloud project 'apache-beam-testing'
. Each instance has 16 CPUs and 104GB of memory. We used to run Jenkins nodes using Apache Jenkins Build cluster (https://builds.apache.org/), but we have since migrated to our dedicated cluster: https://ci-beam.apache.org/, and there is an old group of workers 'apache-beam-jenkins-jnlp'
that is now stopped but not yet deleted.
Instance Name | Jenkins agent name | Jenkins Label |
---|---|---|
apache-ci-beam-jenkins-1 | apache-beam-jenkins-1 | beam |
apache-ci-beam-jenkins-2 | apache-beam-jenkins-2 | beam |
apache-ci-beam-jenkins-3 | apache-beam-jenkins-3 | beam |
apache-ci-beam-jenkins-4 | apache-beam-jenkins-4 | beam |
apache-ci-beam-jenkins-5 | apache-beam-jenkins-5 | beam |
apache-ci-beam-jenkins-6 | apache-beam-jenkins-6 | beam |
apache-ci-beam-jenkins-7 | apache-beam-jenkins-7 | beam |
apache-ci-beam-jenkins-8 | apache-beam-jenkins-8 | beam |
apache-ci-beam-jenkins-9 | apache-beam-jenkins-9 | beam |
apache-ci-beam-jenkins-10 | apache-beam-jenkins-10 | beam |
apache-ci-beam-jenkins-11 | apache-beam-jenkins-11 | beam |
apache-ci-beam-jenkins-12 | apache-beam-jenkins-12 | beam |
apache-ci-beam-jenkins-13 | apache-beam-jenkins-13 | beam |
apache-ci-beam-jenkins-14 | apache-beam-jenkins-14 | beam |
apache-ci-beam-jenkins-15 | apache-beam-jenkins-15 | beam |
apache-ci-beam-jenkins-16 | apache-beam-jenkins-16 | beam-perf |
Current installations
- apache-maven:3.5.4
- Docker:18.09.4
- Go:1.16
- kubectl
- open-jdk-11
- open-jdk-1.8
- Pip:8.1.1
- Python:2.7.12
- Python:3.5.2
- Python:3.6.13
- Python:3.7.10
- Python:3.8.9
- Python:3.9.4
Virtualenv:16.4.3
How to install and upgrade software on Jenkins workers
All software updates must be performed and verified in a temporary experimental compute instance. To install or upgrade tools for Jenkins instances:
Create a compute instance with the most recently created boot image.
gcloud compute instances create $USER-jenkins-upgrade --project=apache-beam-testing --zone us-central1-b --image-family=jenkins-worker-boot-image --machine-type=n1-highmem-16
You can create the instance through VM Instances in Cloud Console or by running the
gcloud
command.Connect using ssh to the instance via Cloud Console or
gcloud
command.gcloud compute --project "apache-beam-testing" ssh --zone "us-central1-b" $USER-jenkins-upgrade
Login as 'jenkins' user.
sudo su jenkins
- Install or upgrade the required tools.
Verify the tool by running corresponding Beam tests.
Verify backwards compatibility by running Beam PostCommits against all SDKs.
Clone Beam repository from Git:
git clone https://github.com/apache/beam.git && cd beam
Run your tests by Gradle wrapper. For example:
./gradlew :PythonPostCommit --continue --<args>
Remove beam repository after verification.
Create a new image from the your instance and name it as
jenkins-worker-boot-image-YYYYMMDD
. Put the image in the family of'jenkins-worker-boot-image'
, and add a description of your changes.gcloud compute instances stop $USER-jenkins-upgrade --project "apache-beam-testing" --zone "us-central1-b" gcloud compute images create \ --project apache-beam-testing \ --source-disk $USER-jenkins-upgrade \ --source-disk-zone us-central1-b \ jenkins-worker-boot-image-<YYYYMMDD> \ --family jenkins-worker-boot-image \ --description "Updated XYZ, see BEAM-1234."
To reboot Jenkins executors with a new image, do the following:
- Go to the page for the Jenkins agent you want to update (for example) and click "Mark this node temporarily offline", leaving a reason such as "Updating X dependency."
- Wait until there are no more tests running in that agent (under "Build Executor Status" on the left of the page). You can also prioritize the update on agents currently not running any builds.
- Restart the VM with a new disk:You could also do these steps via Cloud Console instead of CLI:
OLD_IMAGE_DATE=20200729 NEW_IMAGE_DATE=20211011 PROJECT=apache-beam-testing ZONE=us-central1-b # Select the workers currently offline and not processing any builds, # that you want to upgrade. # For example: for CI_INSTANCE_NUMBER {1..4} {10..12} 16; do for CI_INSTANCE_NUMBER in {1..4}; do CI_INSTANCE=apache-ci-beam-jenkins-${CI_INSTANCE_NUMBER} gcloud compute instances stop $CI_INSTANCE --project $PROJECT --zone $ZONE gcloud compute instances detach-disk $CI_INSTANCE --project $PROJECT --zone $ZONE --disk ${CI_INSTANCE}-${OLD_IMAGE_DATE} gcloud compute disks create ${CI_INSTANCE}-${NEW_IMAGE_DATE} --project $PROJECT --zone $ZONE --image=jenkins-worker-boot-image-${NEW_IMAGE_DATE} gcloud compute instances attach-disk $CI_INSTANCE --project $PROJECT --zone $ZONE --disk ${CI_INSTANCE}-${NEW_IMAGE_DATE} --device-name=${CI_INSTANCE}-${NEW_IMAGE_DATE} --boot gcloud compute instances start ${CI_INSTANCE} --project $PROJECT --zone $ZONE done
Stop the VM, then click edit.
Delete the existing boot disk by clicking the 'X'.
Create a new boot disk by using the newly generated image.
Name disk as
'apache-ci-beam-jenkins-[1,16]-YYYYMMDD'
.Make sure the boot disk 'Model' is 'Boot, read/write'.
Start the VM.
- Mark the node in Jenkins as online again and click "Launch agent" button if there is one.
- Verify that inventory job for this node did not start while the worker was offline, sample build for node #10: https://ci-beam.apache.org/job/beam_Inventory_apache-beam-jenkins-10. If there is a stuck execution of the inventory job, cancel it manually. TODO (BEAM-13666): we should cancel inventory stuck jobs automatically.
- (Optional) If this is the first agent being updated and you would like to test your changes by running a particular Jenkins job, do the following:
- Go to the page for the job you want to run (example).
- Click "Configure" on the left menu.
- Find the checkmark "Restrict where this project can be run" and change the restriction from "beam" to the specific name of the agent (ex. "apache-beam-jenkins-1").
- Save and apply that change.
- Back on the page for the job, click "Build with Parameters" on the left menu.
- Run the build on "master".
- Once you're done checking the results, change the restriction for the job back to "beam".
- Repeat for each worker. You can also do it in batches of workers, but avoid bringing all of the workers offline at once.
After confirming that update was successful, we can go to Cloud Console Disks and clean up deprecated Jenkins worker disks, or do this via cli:
OLD_IMAGE_DATE=20200729 PROJECT=apache-beam-testing ZONE=us-central1-b for CI_INSTANCE_NUMBER in {1..16}; do CI_INSTANCE=apache-ci-beam-jenkins-${CI_INSTANCE_NUMBER} gcloud compute disks delete ${CI_INSTANCE}-${OLD_IMAGE_DATE} --project $PROJECT --zone $ZONE --quiet done