THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
...
| Naming | Tagging | Example | |
|---|---|---|---|
| Snapshot image | language + language_version | yyyymmdd-hhmmss _{status} in UTC | gcr.io/apache-beam-testing/beam/sdk/java8:20190820-020000/snapshot/yyyy/mm/dd/java:20190820_verified |
| Release images | language + language_version (Java and Go go without language version until we support it.) | Beam release version | gcr.io/apache-beam-testing/beam/sdk/release/python2.7:2.10.1 |
Gcr.io
Things to know about gcr.io:
...
To build docker images run:
| Code Block | ||||||
|---|---|---|---|---|---|---|
| ||||||
$ pwd ...beam/ $ ./gradlew :sdks:python:container:py27:docker :sdks:python:container: |
...
py35:docker :sdks:python:container:py36:docker :sdks:python:container:py37:docker :sdks:java:container:docker :sdks:go:container:docker --info |
This produces local images named $
| Code Block | ||||||
|---|---|---|---|---|---|---|
| ||||||
${USER}-docker-apache.bintray.io/beam/python2.7 ${USER}-docker-apache.bintray.io/beam/python3.5 ${USER}-docker-apache.bintray.io/beam/python3 |
...
.6 ${USER}-docker-apache.bintray.io/beam/ |
...
python3.7${USER}-docker-apache.bintray.io/beam/java |
...
${USER}-docker-apache.bintray.io/beam/go |
You can set the repository root and tag when building with -Pdocker-repository-root= and -Pdocker-tag= on the gradle command line.
They can be examined by running docker commands:
...
| Code Block | ||||||
|---|---|---|---|---|---|---|
| ||||||
$ docker image ls
|
...
$ docker run --entrypoint bash -it ${USER}-docker-apache.bintray.io/beam/python3:latest |
Run a test against locally built container
Running dataflow tests with a custom container requires GCR and dataflow permissions in a cloud project and a GCS location. By default these are apache-beam-testing and gs://temp-storage-for-end-to-end-tests but these can be overridden with PROJECT and GCS_LOCATION environment variables. $ bash
| Code Block | ||||||||
|---|---|---|---|---|---|---|---|---|
| ||||||||
$ bash sdks/python/container/run_validatescontainer.sh |
...
python2 $ bash sdks/python/container/run_validatescontainer.sh python3.5 $ bash sdks/python/container/run_validatescontainer.sh python3.6 $ bash sdks/python/container/run_validatescontainer.sh python3 |
...
.7 |
...
| Code Block | ||||||||
|---|---|---|---|---|---|---|---|---|
|
$ ./gradlew :javaPostCommitPortabilityApi --continue --info |
...
| Code Block | ||||||||
|---|---|---|---|---|---|---|---|---|
| ||||||||
$ ./gradlew :goPostCommit |
Running precommit tests.
| Code Block | ||||||||
|---|---|---|---|---|---|---|---|---|
| ||||||||
$ ./gradlew :sdks:python:test-suites:portable:py2:preCommitPy2
$ ./gradlew :sdks:python:test-suites:portable:py35:preCommitPy35
$ ./gradlew :sdks:python:test-suites:portable:py35:preCommitPy36
$ ./gradlew :sdks:python:test-suites:portable:py35:preCommitPy37 |
| Code Block | ||||||||
|---|---|---|---|---|---|---|---|---|
| ||||||||
$ ./gradlew :javaPreCommitPortabilityApi --continue --info |
| Code Block | ||||||||
|---|---|---|---|---|---|---|---|---|
| ||||||||
$ ./gradlew :goPreCommit |
TODO: how to run against the prebuilt container?
...
Publish
Publishing an image to gcr.io/beam requires permissions in apache-beam-testing project.
...