Page History

Starting with Solr 9, we have changed our build process to be Gradle, Ant is no longer supported. Versions of Solr through Solr 8x are built only with Ant. This page is intended to help people use Gradle to build Solr, with special attention to tips for people already familiar with the Ant build process.

Solr versions through 8x must be built with Ant.

I want to especially thank Dawid Weiss for all his work on this, without his help and guidance (ok, his willingness to do most of the heavy lifting), we wouldn't be able to even consider transitioning to Gradle yet or even in the foreseeable future.

...

JIRA:  SOLR-13452 Update the lucene-solr build from Ivy+Ant+Maven (shadow build) to Gradle.

Project Checkout and Setup

Let's check out the project and switch to a Gradle build branch. There is no need to install Gradle or worry about the version - the right version will be automatically installed for you.

Setup

If you get through these steps, you're ready to start using Gradle regularly:

1. Get a current version of Lucene/Solr with the usual "git clone https://github.com/apache/lucene-solr.git _lucene-solr dest"
2. cd _lucene-solr
3. git checkout -t origin/jira/SOLR-13452_gradle_7
4. gw eclipse
5. In Eclipse, File→Import→Import Existing Projects into Workspace, select the root of the project directory we just checked out, choose to include nested projects.

Image Removed

I also recommend adding a few properties to ~/.gradle/gradle.properties for a better build experience.

You can set recommended defaults with:

gw defaultUserConfig

You can get more aggressive defaults for better build performance at the cost of resources with:

gw defaultUserConfig --style=aggressive

Some Basics

Let's try out a few basics.

1. gw tasks                            // list the available tasks we can run
2. gw clean build -x test  // build everything and run checks, but skip unit tests
3. gw clean build                // do it again with unit tests
4. gw compileJava             // just compile java source code
5. gw compileTestJava    // just compile java test source code
6. gw clean test --tests org.apache.solr  // run a single test - do not use wildcards or every test will be examined to find the one test
7. gw clean test --tests *HDFS*Test  // run a subset of tests
8. gw check                          // runs various checks like forbiddenApis, illegal source patterns, license headers, etc - depends on test task and is run as part of build task
9. cd lucene; gw test        // run the Lucene tests

The root of the build is at _lucene-solr/build.gradle. This is where common things that are applied to multiple sub projects across Lucene and Solr live. This is where you might exclude a dependency from the entire project or add a task for all of the projects.

Each sub project will then have it's own build.gradle where things are customized or added and module dependencies are defined (without versions, as they come from a common root versions file).

There are also a few aggregate projects, for example, _lucene-solr/solr/build.gradle is where the Solr packageDist task lives. A task by the same name is in the Lucene aggregate project's build.gradle. These aggregate projects do not produce standard Java artifacts or get published.

Notes

• All versions for dependencies except those used in buildSrc are specified in the root versions.props file.
• You can see the actual dependencies and versions that get pulled in in the root versions.lock file.
• The buildSrc directory is a special directory to Gradle that is it's own root project that can host Gradle plugins that we can access from our root project.
• We also abuse buildSrc as a subproject of our root project to share resources.

Configuration

Project configuration is in _lucene-solr/gradle.properties. This is where our project defaults live and they can be overridden for your local env in ~/.gradle/gradle.properties or via the command line.

gradle.properties:

version=9.0.0 // the project version

// whether or not to use the Gradle daemon - if true, keeps the build process around for reuse for up to 3 hours so that startup times are removed and hotspot has a chance to rock - you should generally set this to true in your ~/.gradle/gradle.properties
// we set to false so that a CI system with no setting will not use a daemon by default
org.gradle.daemon=false

// allows tasks to be executed in parallel, across modules
org.gradle.parallel=true

// max parallel jobs to run at once, including both tasks and tests.
// default is number of CPU cores which is often too high - you should start by setting it to half the number of cores, especially if you have hyperthreading
# org.gradle.workers.max=2

// number of jvms to distribute tests across in parallel
// defaults to number of CPU cores / 2 - you should just set this the same as org.gradle.workers.max
// NOTE: gradle does not try to balance tests across jvms yet: https://github.com/gradle/gradle/issues/2669
# tests_jvms=5

// enables gradles build cache, which will reuse cached build outputs when it can
org.gradle.caching=true

// experimental gradle feature - does not currently work with our version constraints plugin: https://github.com/palantir/gradle-consistent-versions/pull/145
// also known to have other issues and not known to really speed anything up anyhow
org.gradle.configureondemand=false

// how much ram the gradle daemon or process can use
org.gradle.jvmargs=-Xmx1g

More Advanced

Moving beyond the basics.

1. gw regenerate                       // rebuilds all of our generated source and resource files - doing this for every module requires some dependencies, you can see some hints for that here
2. gw publishToMavenLocal  // publish all modules to local ~/.m2 Maven folder
3. gw packageDist                    // generate packaged distributions for Lucene and Solr in their respective dist directories
4. cd solr; gw packageDist    // generate packaged distributions for Solr

Dependencies

TBD

Pointers

1. dest
2. ./gradlew help. Gradlew is "gradle wrapper" and should automatically download gradle and start it running.
3. ./gradlew helpAnt - For people already familiar with building Solr with Ant, this command will show Gradle tasks that are equivalent to selected Ant targets.
4. The Gradle project is here if you have trouble, or ping the Lucene dev list:  https://docs.gradle.org/current/userguide/installation.html
5. ./help contains plain-text files you can scan, for instance 'ant.txt' is the output from `./gradlew helpAnt`.

Useful commands:

Here are some commands that will get you started (all prefixed by ./gradlew or just gw if you've installed gdub).

• help - of course.
• helpAnt - lists Gradle commands that are the equivalent of what you may be used to in Ant.
• assemble -  will create a runnable Solr instance in ./solr/packaging/build/solr-#.#.#-SNAPSHOT. However, it will delete that directory next time it's run.
• dev - like assemble, but will write the output code to ./solr/packaging/build/dev and will not delete the directory first. Use this if you're developing code, instantiating Solr and recompiling and need to preserve your setup.
• tasks - lists all the tasks you can execute. There are a lot of them...
• check - will run all of the checks (ant precommit+) and run all of the tests. Please get in the habit of running this! More below for why in "Differences from Ant".
  • Adding -x test will run all the validation checks but not run tests (this is the rough equivalent of ant precommit)

The first time you run these commands, all of the dependencies for Solr will be downloaded to (*nix) ~/.gradle.caches, and this process may take quite some time depending on your network connectivity.

Differences from Ant:

Some operations that are simply different from Ant. Here are some hints to help with these:

• Gradle runs fastest when it uses a daemon. The first time you use gradle, it'll take some time (usually 10-20 seconds) to start the daemon and build caches. You'll see a message about "starting daemon". Subsequent runs should mostly skip this step
  • You may want to change org.gradle.daemon.idletimeout=900000 in your gradle.properties file (written the first time you run gradle) to a longer interval (this is in ms) to avoid restarting the daemon as frequently.
  • gradlew --stop will stop the daemon if you think it's confused.
• The build will fail on un-suppressed warnings. Trunk compiles cleanly, all warnings are suppressed or have been fixed (mostly suppressed). The first option if your new code generates more warnings is to change the code so it doesn't. Failing that add a SuppressWarnings annotation, but please try to fix the code first.
  • If you're working on code and can remove some of these suppressions, please do. It was too big a task to try to fix all the warnings by fixing the code all at once. We should improve from here.
• The validation checks fail on log messages that follow potentially wasteful patterns. ./gradlew helpValidateLogCalls describes how to avoid these errors. This is part of ./gradlew check.
  • The validation is a bit harsh, if you have a logging call that is flagged but you've inspected and know is correct/efficient, add //logok as a comment. But please read through ./gradlew helpValidateLogCalls for guidelines, it's more complex than it appears at first glance.
• It's not necessary to run the idea task before opening the project, just "open or import" and choose the root directory of your source tree.
• The output directories have changed for artifacts and test results. Pay close attention to the messages on the screen at the end of a task execution, they'll tell you exactly where to look.

IntelliJ IDE

IntelliJ integration is useful. There's nothing special about our support for Idea, it's just the one that's been exercised most. In particular the gradle window lets you easily execute tasks. In the case of failures because of the -Werror flag and just click to the source. The classes and testClasses tasks are particularly useful here as they just compile the Java code without the extra work the assemble task does.

Running tests in IntelliJ seems to take longer, we're investigating why.

Please do not let IntelliJ "fix" it's inspections by adding the IntelliJ-specific suppression. That said, it's useful to look at the inspections, they may be highlighted. Some of them are not useful, any tips on how to turn off specific ones appreciated. For instance, I personally like if (something == false) because I think if (!something) is easy to misread, but IntelliJ prefers the latter. There's no intention here to make all our code conform to IntelliJs inspections. That said, looking at them can be instructive.

Eclipse IDE:

Eclipse is also a popular IDE, and support has been added. However, this IDE has not been exercised nearly as much as IntelliJ. If you prefere Eclipse, please report any shortcomings on the dev mailing list.

Other IDEs:

Please feel free to add any tips about other IDEs. As mentioned above, IntelliJ was just what has been used most.

Troubleshooting:

We've run into some rough edges that are outlined here:

• When switching back and forth between Gradle on trunk and 8x where you have to use ant, there are some Gradle-specific files that will be on your 8x tree. These should be ignored by the build system and Ant. You can delete them if you want. If you do, be aware that the next time you run Gradle it'll write a new "gradle.properties" file if it's not there so any older edits will be lost.
  
  • Gradle writes a new "gradle.properties" file at the root of your source tree if it's not there already, sometimes this can surprise you
• I've occasionally seen gradle stop with a message about thrashing memory when running the check task. There are two short-term solutions here: 
  • ./gradlew --stop will halt the daemon, releasing memory
  • Increase the memory allocated to Gradle by editing the settings in your gradle.properties file.
  • On the TODO list is to figure out why 3G (the current default which I sometimes bump even higher) and address whatever is causing that.
• A Docker based test framework for the build itself: https://github.com/apache/lucene-solr/tree/jira/SOLR-13452_gradle_7/buildSrc/build-wdocker-test
• The configuration files for IDE's: https://github.com/apache/lucene-solr/tree/jira/SOLR-13452_gradle_7/buildSrc/ide
• Common Gradle config for tests, repositories, build help: https://github.com/apache/lucene-solr/tree/jira/SOLR-13452_gradle_7/buildSrc/common
• Our project and source checkers: https://github.com/apache/lucene-solr/tree/jira/SOLR-13452_gradle_7/buildSrc/src/main/groovy/org/apache/lucene/gradle/checks
• Our logic for packaging releases: https://github.com/apache/lucene-solr/tree/jira/SOLR-13452_gradle_7/buildSrc/src/main/groovy/org/apache/lucene/gradle/dist

Related articles

...