Let's try out the new Lucene/Solr Gradle build!

JIRA:  SOLR-13452 Update the lucene-solr build from Ivy+Ant+Maven (shadow build) to Gradle.


Before we begin, I recommend using https://github.com/dougborg/gdub as this lets you run targets from submodules rather than having to do everything top level with gradlew.
Where you might normally do ./gradlew solr:core:test from the root directory, gdub lets you do cd solr/core; gw test.


Project Checkout and Setup

Let's check out the project and switch to a Gradle build branch. There is no need to install Gradle or worry about the version - the right version will be automatically installed for you.

  1. git clone https://github.com/apache/lucene-solr.git _lucene-solr
  2. cd _lucene-solr
  3. git checkout -t origin/jira/SOLR-13452_gradle_8
  4. gw eclipse
  5. In Eclipse, File→Import→Import Existing Projects into Workspace, select the root of the project directory we just checked out, choose to include nested projects.


I like to check out the project in a directory that starts with underscore (_lucene-solr). Eclipse is very finicky about wanting the root project name to match it's folder name, so this saves some hassle while causing the root project to show up first in Eclipse (see below).

This is nice behavior because as you can see, our non Lucene/Solr specific projects come first (_lucene-solr, buildSrc, dev-tools), then we have the Lucene projects and then the Solr projects.


I also recommend adding a few properties to ~/.gradle/gradle.properties for a better build experience.

You can set recommended defaults with:

gw defaultUserConfig

You can get more aggressive defaults for better build performance at the cost of resources with this command (you may have to create a ~/.gradle/gradle.properties first):

gw defaultUserConfig --style=aggressive

Some Basics

Let's try out a few basics.

  1. gw tasks                            // list the available tasks we can run
  2. gw clean build -x test  // build everything and run checks, but skip unit tests
  3. gw clean build                // do it again with unit tests
  4. gw compileJava             // just compile java source code
  5. gw compileTestJava    // just compile java test source code
  6. gw clean test --tests org.apache.solr  // run a single test - do not use wildcards or every test will be examined to find the one test
  7. gw clean test --tests *HDFS*Test  // run a subset of tests
  8. gw check                          // runs various checks like forbiddenApis, illegal source patterns, license headers, etc - depends on test task and is run as part of build task
  9. cd lucene; gw test        // run the Lucene tests


The root of the build is at _lucene-solr/build.gradle. This is where common things that are applied to multiple sub projects across Lucene and Solr live. This is where you might exclude a dependency from the entire project or add a task for all of the projects.

Each sub project will then have it's own build.gradle where things are customized or added and module dependencies are defined (without versions, as they come from a common root versions file).

There are also a few aggregate projects, for example, _lucene-solr/solr/build.gradle is where the Solr packageDist task lives. A task by the same name is in the Lucene aggregate project's build.gradle. These aggregate projects do not produce standard Java artifacts or get published.

Notes

Configuration

Project configuration is in _lucene-solr/gradle.properties. This is where our project defaults live and they can be overridden for your local env in ~/.gradle/gradle.properties or via the command line.

gradle.properties:

// the project version

version=9.0.0 

// whether or not to use the Gradle daemon - if true, keeps the build process around for reuse for up to 3 hours so that startup times are removed and hotspot has a chance to rock - you should generally set this to true in your ~/.gradle/gradle.properties
// we set to false so that a CI system with no setting will not use a daemon by default
org.gradle.daemon=false

// allows tasks to be executed in parallel, across modules
org.gradle.parallel=true

// max parallel jobs to run at once, including both tasks and tests.
// default is number of CPU cores which is often too high - you should start by setting it to half the number of cores, especially if you have hyperthreading
# org.gradle.workers.max=2

// number of jvms to distribute tests across in parallel
// defaults to number of CPU cores / 2 - you should just set this the same as org.gradle.workers.max
// NOTE: gradle does not try to balance tests across jvms yet: https://github.com/gradle/gradle/issues/2669
# tests_jvms=5

// enables gradles build cache, which will reuse cached build outputs when it can
org.gradle.caching=true

// experimental gradle feature - does not currently work with our version constraints plugin: https://github.com/palantir/gradle-consistent-versions/pull/145
// also known to have other issues and not known to really speed anything up anyhow
org.gradle.configureondemand=false

// how much ram the gradle daemon or process can use
org.gradle.jvmargs=-Xmx1g

More Advanced

Moving beyond the basics.

  1. gw regenerate                       // rebuilds all of our generated source and resource files - doing this for every module requires some dependencies, you can see some hints for that here
  2. gw publishToMavenLocal  // publish all modules to local ~/.m2 Maven folder
  3. gw packageDist                    // generate packaged distributions for Lucene and Solr in their respective dist directories
  4. cd solr; gw packageDist    // generate packaged distributions for Solr

Dependencies

TBD

Pointers

The awkard period of using both Ant and Gradle

Until we work all the kinks out, we need to be able to use both Gradle and Ant builds. As soon as Gradle does everything we need, we'll remove the Ant option. Currently there are a couple of "gotchas".

Related articles

Related articles appear here based on the labels you select. Click to edit the macro and add or change labels.



Related issues