You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 21 Next »

Gradle build files are written using Apache Groovy.

Gradle Tests Parallelization

The build is configured to run on a given number of workers that are responsible for all the tasks of the build (running tests, compiling code, checkstyle, findbugs, ...)

Gradle is configured with the forkEvery parameter to run test classes in parallel among the gradle workers.

Each gradle worker uses a single JVM. Inside it, tests are run in sequence. Tests inside a test class are not run in parallel unless you specify a JUnit parallelized runner.


Specifying build targets

Gradle targets can be specified with either the project name or directory, for example:

$ ./gradlew :website:serveWebsite

is the same as

$ ./gradlew -p website serveWebsite

The mapping of directory to project name is in settings.gradle

Code Coverage

Because of issues with the Jacoco plugin, it is disabled unless explicitly requested. And if you request a report, you must also request the tests to report on, like so:

./gradlew -p sdks/java/extensions/sql test jacocoTestReport

Troubleshooting build issues

You can increase the log level of the build with command-line flags:

  • --info flag adds verbose logging
  • --stacktrace flag adds a stacktrace on failures
  • --scan flag creates a Gradle Build Scan


There are also various built-in tasks useful for introspecting components of the build:

  • projects: Lists all projects in present in the build
  • tasks: Displays tasks runnable from the root project (including those in subprojects)
  • dependencies: Displays dependencies for a project, for each configuration scope
  • dependencyInsight --dependency <name> --configuration <name>: Inspect why a dependency exists for a project configuration
  • properties: Print the properties of a configured project
  • help --task <taskName>: Display help message for a specific task.

Visualizing Task Dependencies

Our build uses the gradle-task-tree plugin to visualize execution graph task dependencies during the build. This makes it easy to inspect why a task was included and ordering constraints.

To produce a dependency graph, invoke Gradle with a set of tasks and include taskTree from the same project (i.e. ./gradlew :beam-sdks-java-core:build :beam-sdks-java-core:taskTree). Instead of executing the tasks, the gradle-task-tree plugin will print out an ASCII dependency tree on the commandline.

Troubleshooting Resources

Common build logic

Most re-usable steps are in BeamModulePlugin.groovy.

A common style in the build files for methods with several arguments is to use the named argument constructor which allows passing a map for names and values, for example:

class BuildTaskConfiguration {
  String name
  boolean useTestConfig = false
  String baseUrl = ''
  String dockerWorkDir = ''
}

def createBuildTask = {
  BuildTaskConfiguration config = it as BuildTaskConfiguration
...
}

createBuildTask(name:'Local', dockerWorkDir: dockerWorkDir)

Customizing local build configuration

Our Gradle uses various configuration properties which can be overridden to customize build behavior. These are generally used to set external resource locations for testing, such as Google Cloud project or Docker container registry. You can find useful properties by searching for usages of the Gradle findProperty('..') API.

You can specify property values for these on the command-line using -P. For example, to run Dataflow runner precommit tests with your own GCP settings:

./gradlew -PgcpProject=myGoogleCloudProject -PgcsBucket=myGoogleCloudStorageBucket \
:runners:google-cloud-dataflow-java:examples:preCommit

Setting build customizations in a user config file

You can also set default Gradle property overrides inside of a user gradle.properties file (documentation). Create the file in your GRADLE_USER_HOME directory (defaults to ~/.gradle on Mac/Linux). In the file, you can set defaults for any Gradle property from the build:

gcsProject=myGoogleCloudProject
gcpProject=myGoogleCloudProject
gcsTempRoot=myGoogleCloudStorageBucket
dockerImageRoot=myDockerImageRoot
docker-repository-root=myDockerRepositoryRoot

Create Build Scan on failed builds

Another useful customization is to automatically upload a build scan when the build fails. This makes it very easy to share your build results for debugging. To do so, create an init.d/buildScan.gradle file in your GRADLE_USER_HOME directory with the following contents (documentation):

initscript {
  repositories {
    gradlePluginPortal()
  }
    dependencies {
    classpath 'com.gradle:build-scan-plugin:2.4.2'
  }
}
rootProject {
  apply plugin: com.gradle.scan.plugin.BuildScanPlugin
  buildScan {
    publishOnFailure()
    termsOfServiceUrl = 'https://gradle.com/terms-of-service'
    termsOfServiceAgree = 'yes'
  }
}

OutOfMemoryException during the Gradle build

You may want to have a customized gradle.properties file in GRADLE_USER_HOME, and then setup your environment vars in the shell startup script. 

export GRADLE_USER_HOME=~/.gradle

source ~/.bashrc


Update gradle config via editing/creating gradle.properties in root source folder or in GRADLE_USER_HOME (~/.gradle).

cat ~/.gradle/gradle.properties 

# Gradle has a tendency to reuse daemons during build.
# Disable daemon usage to avoid OOM.
org.gradle.daemon=false


# By default Beam project is configured to run gradle tests in parallel. Disabling this can reduce max memory usage.
org.gradle.parallel=false


# Increase jvm memory.
org.gradle.jvmargs=-Xmx2g -XX:MaxPermSize=512m

Publish a module snapshot release to mavenLocal

For local testing, a module can be published to mavenLocal. E.g. for the kafka module:

./gradlew -Ppublishing -PdistMgmtSnapshotsUrl=~/.m2/repository/ -p sdks/java/io/kafka publishToMavenLocal

If the local maven repository isn't in the default path of ~/.m2/repository/ then it can be replaced with a custom path like the following: file:///path/to/custom/maven/repo

Once a snapshot version of Beam has been built, you must depend on that snapshot when running pipelines. See the snippet below for an example of how to use the snapshot as a dependency in Gradle. For more info see the Gradle documentation.

repositories {
   maven {
       url "file:///path/to/custom/maven/repo"
   } // Use this when using a non-default local repo.
   mavenLocal() // Use this with the default local repo.
   mavenCentral()
}

dependencies {
   compile "org.apache.beam:beam-sdks-java-io-kafka:2.23.0-SNAPSHOT"
}

Publish a vendored dependency locally

./gradlew -p vendor/bytebuddy-1_10_8 -PisRelease -Psigning.gnupg.keyName=apache.org -PvendoredDependenciesOnly publishToMavenLocal

You can find this and more details in the vendored dependencies release guide.

Copying Test Resources Across Gradle Projects

For tests run in one project (e.g. integration tests run by :sdks:java:io:google-cloud-platform) that run tests defined in another (e.g. :runners:google-cloud-dataflow-java) your tests may depend on resources (canonically in src/test/resources of that source tree).
To add these files to the build.gradle to include a copy task to get files from one project to another.

task copyGoogleCloudPlatformTestResources(type: Copy){
  from project(':sdks:java:io:google-cloud-platform').fileTree("src/test/resources")
  into "$buildDir/resources/test/"
}

task googleCloudPlatformLegacyWorkerIntegrationTest(type: Test, dependsOn: copyGoogleCloudPlatformTestResources) {
  ...}
  • No labels