...
| Code Block |
|---|
| language | bash |
|---|
| title | Python |
|---|
| linenumbers | true |
|---|
|
$ pwd
[...]/beam/
$ ./gradlew :sdks:python:container:py2:docker -Pdocker-repository-root=$REPOSITORY -Pdocker-tag=$TAG --info
$ ./gradlew :sdks:python:container:py35:docker -Pdocker-repository-root=$REPOSITORY -Pdocker-tag=$TAG --info
$ ./gradlew :sdks:python:container:py36:docker -Pdocker-repository-root=$REPOSITORY -Pdocker-tag=$TAG --info
$ ./gradlew :sdks:python:container:py37:docker -Pdocker-repository-root=$REPOSITORY -Pdocker-tag=$TAG --info
|
...
Run a test against locally built container
...
Please note all following tests create new images each time, so they don't use the images we created above, however, since they are created from the same code, so we assume images are exactly same and if all tests passed with the new created images then assume images created above also pass the tests.
Running precommit tests at local.
| Code Block |
|---|
| language | bash |
|---|
| title | Python |
|---|
| linenumbers | true |
|---|
|
$ bash$ ./gradlew :sdks/python/container/run_validatescontainer.sh python2
$ bash sdks/python/container/run_validatescontainer.sh python3.5
$ bash sdks/python/container/run_validatescontainer.sh python3.6
$ bash sdks/python/container/run_validatescontainer.sh python3.7:python:test-suites:portable:py2:preCommitPy2
$ ./gradlew :sdks:python:test-suites:portable:py35:preCommitPy35
$ ./gradlew :sdks:python:test-suites:portable:py35:preCommitPy36
$ ./gradlew :sdks:python:test-suites:portable:py35:preCommitPy37 |
| Code Block |
|---|
| language | bash |
|---|
| title | Java |
|---|
| linenumbers | true |
|---|
|
$ ./gradlew :javaPostCommitPortabilityApijavaPreCommitPortabilityApi --continue --info |
| Code Block |
|---|
| language | bash |
|---|
| title | Go |
|---|
| linenumbers | true |
|---|
|
$ $ ./gradlew :goPostCommitgoPreCommit |
...
Run postcommit tests at local
...
:
| Code Block |
|---|
| language | bash |
|---|
| title | Python |
|---|
| linenumbers | true |
|---|
|
$ ./gradlew :sdks:python:test-suites:portable:py2:preCommitPy2python2PostCommit
$ ./gradlew :sdks:python:test-suites:portable:py35:preCommitPy35python35PostCommit
$ ./gradlew :sdks:python:test-suites:portable:py35:preCommitPy36python36PostCommit
$ ./gradlew :sdks:python:test-suites:portable:py35:preCommitPy37 |
| Code Block |
|---|
| language | bash |
|---|
| title | Java |
|---|
| linenumbers | true |
|---|
|
$ ./gradlew :javaPreCommitPortabilityApijavaPostCommitPortabilityApi --continue --info |
| Code Block |
|---|
| language | bash |
|---|
| title | Go |
|---|
| linenumbers | true |
|---|
|
$ $ ./gradlew :goPreCommitgoPostCommit |
Publish
Publishing an image to gcr.io/beam requires permissions in apache-beam-testing project.
...
This section describes how to validate a built and/or published image.
| Code Block |
|---|
| language | bash |
|---|
| title | Make sure the images are pullable |
|---|
| linenumbers | true |
|---|
|
$ docker pull $REPOSITORY/python2.7:$TAG
$ docker pull $REPOSITORY/java:$TAG
$ docker pull $REPOSITORY/go:$TAG |
Automated test suites
We have these test suites in Beam that utilize portability:
...
$ ./gradlew ... ?
Manual testing
To run a custom pipeline 
against an image :
...
Use uploaded images and test on Dataflow.
| Code Block |
|---|
| language | bash |
|---|
| title | Python |
|---|
| linenumbers | true |
|---|
|
# this should run againest py2.7, py3.5, py3.6 and py3.7.
$ pwd
[...]beam//sdks/python
$ python -m apache_beam.examples.wordcount \
--input gs://apache-beam-samples/shakespeare/hamlet.txt \
--output gs://temp-storage-for-end-to-end-tests/staging-$USER/output \
--runner DataflowRunner \
--project apache-beam-testing \
--temp_location gs://temp-storage-for-end-to-end-tests/staging-$USER/ \
--worker_harness_container_image $REPOSITORY/python2.7:$TAG \
--experiment beam_fn_api \
--sdk_location sdks/python/container/py2/build/target/apache-beam.tar.gz |
| Code Block |
|---|
| language | bash |
|---|
| title | Java |
|---|
| linenumbers | true |
|---|
|
// get WordCount example code as a maven project
$ mvn archetype:generate \
-DarchetypeGroupId=org.apache.beam \
-DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples \
-DarchetypeVersion=2.6.0 \
-DgroupId=org.example \
-DartifactId=word-count-beam \
-Dversion="0.1" \
-Dpackage=org.apache.beam.examples \
-DinteractiveMode=false
// run Java project
$ pwd
[...]/word-count-beam
$ mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount -Dexec.args="\
--runner=DataflowRunner \
--project=apache-beam-testing \
--stagingLocation=gs://temp-storage-for-end-to-end-tests/staging-$USER/ \
--workerHarnessContainerImage=$REPOSITORY/java:$TAG \
--experiments=beam_fn_api \
--output=gs://temp-storage-for-end-to-end-tests/staging-$USER/output" \
-Pdataflow-runner |
| Code Block |
|---|
| language | bash |
|---|
| title | Go |
|---|
| linenumbers | true |
|---|
|
$ pwd
[...]/beam/sdks/go
$ go run examples/wordcount/wordcount.go \
--runner=dataflow \
--project=apache-beam-testing \
--staging_location=gs://temp-storage-for-end-to-end-tests/staging-$USER/ \
--worker_harness_container_image=$REPOSITORY/go:$TAG \
--output=gs://temp-storage-for-end-to-end-tests/staging-$USER/output |
Backwards compatibility
- Do we want new images to be able to run old pipelines?
- Vice-versa?
- How long do we support backwards compatibility for?
...
