Please visit Google Doc for review, wiki page will be updated after release plan is finalized.
Table of Contents |
---|
This page describes the release, validation, and other aspects related to SDKHarness images.
...
This section describes the naming scheme and location for publication of the containersimages.
Proposed repository
We propose to use gcr.io/beam, created under apache-beam-testing project, artifacts accessible publicly. SDK Harness containers go to sdk folder.
Things to know about gcr.io:
- Quotas:
- ?It seems like gcr only has hit limit from each IP, I didn't find any documentation about image limit. But it would have size limit.
- Permissions:
- download access - public
- publish - limited to authorized accounts that have correct permissions under apache-beam-testing:
- publishing the snapshots nightly might be feasible similar to how we currently publish nightly maven snapshots, by creating a Jenkins job;
- publishing at release time can be another Jenkins job that is triggered manually by the release owner;
- what's the process of triggering a job in such case?Depends on how many tests we want to run. If need to run many tests, it should be a Jenkins job, otherwise, we can write a shell script to automate the process.
- GCP Project:
- apache-beam-testing
- If something goes wrong with the release process:
- ping dev@ ?
- If something goes wrong with a customer pipeline from using the prebuilt images:
- ping dev@ ?
Proposed naming and tagging scheme
Naming | Tagging | Example | |
---|---|---|---|
Snapshot images | language + language_version | yyyymmdd_{status} in UTC | gcr.io/apache-beam-testing/beam/sdk/snapshot/2019/08/20/java:20190820_verified |
Release images | language + language_version(Java and Go go without language version until we support multi versions.) | Beam release version | gcr.io/apache-beam-testing/beam/sdk/release/python2.7:2.10.1 gcr.io/apache-beam-testing/beam/sdk/release/java:2.10.1 |
*Java and Go will not have language version until we support multi versions.
Publication Schedule
...
Snapshot images
- how do we publish the snapshots, what's the frequency?
- automatic, when HEAD is built, nightly similar to maven snapshots, or manually?.
- when and how do we cleanup the snapshot images:
- when you publish an image you have a new version/hash and still can use any previous versions. They take space and will count towards a quota. We need to clean them up periodically:
- make it part of the publish job to look and delete the versions that are more than X days (or versions) old?
- when you publish an image you have a new version/hash and still can use any previous versions. They take space and will count towards a quota. We need to clean them up periodically:
...
- How should we build and publish the images for release versions of Beam?
- Start with manually, as part of Beam release , and automate it later.
- Should it be a blocker for the release?
- Should we make it part of the release and not mark release as complete until the images are published?
- For first three release(v2.16 ~ v2.18), we can make it optional, and if all these release go well, we can make it mandatory part of the release(from v2.19).
- Should it be done by the same release owner?
- Yes, this document provides step by step guide about building, testing and pushing. In future, we can automate this process.
- Should the validation be part of the release validation?
- Yes, when we validate code base.
- Should we make it part of the release and not mark release as complete until the images are published?
Docker images
Language | Supported versions | Docker image name |
---|---|---|
Python | 2.7 | python2.7 |
3.5 | python3.5 | |
3.6 | python3.6 | |
3.7 | python3.7 | |
Java | 8 | java |
11 | not available | |
Go | 1.12 | go |
Commands
This section describes the commands that are used to build, publish, run tests and examples for the images.
...
This is the last part of release process, at this moment, we already finished release validation, do we need to run tests against container again which is generated from the already validated code?
Running precommit tests at local
...
Please note this will create new images (not used for above testings). Since they are created from the same code, we assume exactly same and if all tests passed with the new created images then assume images created above also pass the testsassume the images are exactly the same.
Set repository and tag as variables:
...
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
$ pwd [...]/beam/ $ ./gradlew :sdks:python:container:py2:dockerdockerPush -Pdocker-repository-root=$REPOSITORY -Pdocker-tag=$TAG --info $ ./gradlew :sdks:python:container:py35:dockerdockerPush -Pdocker-repository-root=$REPOSITORY -Pdocker-tag=$TAG --info $ ./gradlew :sdks:python:container:py36:dockerdockerPush -Pdocker-repository-root=$REPOSITORY -Pdocker-tag=$TAG --info $ ./gradlew :sdks:python:container:py37:dockerdockerPush -Pdocker-repository-root=$REPOSITORY -Pdocker-tag=$TAG --info |
...
We have these test suites in Beam that utilize portability:
- precommit tests(?)
- postcommit tests(?)tests
*These tests will use default images, not the one pushed to gcr.
Manual testing
Test on Dataflow with uploaded images.
...
- Do we want new images to be able to run old pipelines?
- This is decided by SDK, not container specific.
- How long do we support backwards compatibility for?
- This is decided by SDK, not container specific.
...
- Do we sign the artifacts, images?
- ??
- Do we check hashes, signatures?
- No?
Support Story
How do we allow the images to be used?
- Snapshot images can be used for daily testing.
- Snapshot images are always created from head, so users can use Beam at head if they want to.
- Customize sdks containers on top of published images.
- ...
...
From gCloud instruction, gcr has following quota limitationlimitations.
Info | ||
---|---|---|
| ||
Any request sent to Container Registry has a 2 hour timeout limit. The fixed rate limits per client IP address are:
Container Registry uses Cloud Storage for each registry's underlying storage. Cloud Storage Quotas & Limits apply to each registry. |
...
Any special/extra license needs to be attached to the images?
Report bugs
- If something goes wrong with the release process:
- ping dev@
- If something goes wrong with a customer pipeline from using the prebuilt images:
- ping dev@