FAQ
How do I configure SLF4J?
To configure SLF$J
Add a log4j-test.properties under the directory of the java test.
- Add the following snippets into your
build.gradle
file.test { systemProperty "log4j.configuration", "log4j-test.properties" } dependencies { shadow library.java.slf4j_api shadow library.java.slf4j_log4j12 // or shadow library.java.slf4j_jdk14 }
- The second dependency
shadow library.java.slf4j_log4j12
is not necessary if another library already provides this dependency. Check the dependency included in the dependency tree, execute:
./gradlew dependencies.
Check If you encounter an error message like the following.
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation- If so, it means there is no SLF4J.Add
library.java.slf4j_log4j12
orlibrary.java.slf4j_jdk14
in thebuild.gradle
file.
- If so, it means there is no SLF4J.Add
How to format code automatically and avoid spotless errors?
- Set up a git pre-commit hook to always autoformat code, add the following in
.git/hooks/pre-commit.
- Set the executable bit.
- For more information about git hooks, go to: https://git-scm.com/docs/githooks.
- To skip it, use
--no-verify.
- To disable it, use `
chmod u-x.
How to run a single test?
Execute:
./gradlew :examples:java:test --tests org.apache.beam.examples.subprocess.ExampleEchoPipelineTest --info
How to run Java Dataflow Hello World
pipeline with compiled Dataflow Java worker.
You can dump multiple definitions for a gcp project
name and temp
folder. They are present since different targets use different names.
- Before running the command, configure your gcloud credentials.
- Add
GOOGLE_APPLICATION_CREDENTIALS
to yourenv
variables. - Execute:
./gradlew :runners:google-cloud-dataflow-java:examples:preCommitLegacyWorker -PdataflowProject=<GcpProjectName> -Pproject=<GcpProjectName> -PgcpProject=<GcpProjectName> -PgcsTempRoot=<Gcs location in format: gs://..., no trailing slash> -PdataflowTempRoot=<Gcs location in format: gs://...>
./gradlew :runners:google-cloud-dataflow-java:examples:preCommitFnApiWorker -PdataflowProject=<GcpProjectName> -Pproject=<GcpProjectName> -PgcpProject=<GcpProjectName> -PgcsTempRoot=<Gcs location in format: gs://..., no trailing slash> -PdataflowTempRoot=<Gcs location in format: gs://..., no trailing slash> -PdockerImageRoot=<docker image store location in format gcr.io/...>
How to run a User Defined Pipeline - Java Direct Runner example
If you want to run your own pipeline, and in the meanwhile change beam repo code for dev/testing purposes. Here is an example for a simple runner like directRunner:
- Put your pipeline code under the
example
folder. Add the following build target to the related
build.gradle
:task execute(type:JavaExec) { main = "org.apache.beam.examples.SideInputWordCount" classpath = configurations."directRunnerPreCommit" }
There are also alternative choices, with a slight difference:
Option 1
Create a maven project.
Use the following command to publish changed code to the local repository.
./gradlew -Ppublishing -PnoSigning publishMavenJavaPublicationToMavenLocal
Option 2
- Make use of Integration tests.
- Make your user-defined pipeline part of the integration test.
How to use a snapshot Beam Java SDK version?
To use snapshot BEAM new features prior to the next Beam release, you need to;
- Add the
apache.snapshots
repository to yourpom.xml.
Check this example. - Set
beam.version
to a snapshot version, e.g. "2.24.0-SNAPSHOT" or later (listed here).
Common Errors
Continue on error
Use the --continue
flag makes to compileJava task and to dump all found errors, not stop on first.
IntelliJ Proto Intellisense doesn't work.
This can happen when you start IntelliJ or (in my case) after modifying protos
.
This is not a solved problem yet. But here are some current approaches:
- Clean
build
from console - Build from IntelliJ
- Refresh Gradle Project in IntelliJ
- Restart IntelliJ
- Another option is if
index
is not updated with 3 or 4 steps. For more information, go to Rebuild IntelliJ project indexes.
A workaround that did the trick. Since many things were tried in the process and no clear way to reproduce the error, this might not be the correct or best step. Update steps if you find a shorter or cleaner way to do the trick.
Refresh
tradle
project in IntelliJ.- Close Intellij.
Clean build project from the console. Execute>
./gradlew clean cleanTest build -x testWebsite -x :rat -x test
- Open IntelliJ.
What command should I run locally before creating a pull request?
We recommend running this command, in order to catch common style issues, potential bugs (using code analysis), and Javadoc issues before creating a pull request. Running this takes 5 to 10 minutes.
./gradlew spotlessApply && ./gradlew -PenableCheckerFramework=true checkstyleMain checkstyleTest javadoc spotbugsMain compileJava compileTestJava
If you don't run this locally Jenkins will run them during pre-submit. However, if these fail during pre-submit, you may not see the output of test failures. So doing this first is recommended to make your development process a bit smoother and iterate on your PR until it passes the pre-submit.