THIS IS A TEST INSTANCE. ALL YOUR CHANGES WILL BE LOST!!!!
Reducing Build Times
Spark's default build strategy is to assemble a jar including all of its dependencies. This can be cumbersome when doing iterative development. When developing locally, it is possible to create an assembly jar including all of Spark's dependencies and then re-package only Spark itself when making changes.
| Code Block | ||||
|---|---|---|---|---|
| ||||
$ sbt/sbt clean assemble-deps
$ sbt/sbt package
# ... do some local development ... #
$ sbt/sbt package
# ... do some local development ... #
$ sbt/sbt package
# ...
# You can also use ~ to let sbt do incremental builds on file changes without running a new sbt session every time
$ sbt/sbt ~package |
Checking Out Pull Requests
Git provides a mechanism for fetching remote pull requests into your own local repository. This is useful when reviewing code or testing patches locally. If you haven't yet cloned the Spark Git repository, use the following command:
| Code Block |
|---|
$ git clone https://github.com/apache/spark.git
$ cd spark |
To enable this feature you'll need to configure the git remote repository to fetch pull request data. Do this by modifying the .git/config file inside of your Spark directory. The remote may not be named "origin" if you've named it something else:
| Code Block | ||||
|---|---|---|---|---|
| ||||
[remote "origin"]
url = git@github.com:apache/spark.git
... may be other stuff here ...
fetch = +refs/pull/*/head:refs/remotes/origin/pr/* # Add this line |
Once you've done this you can fetch remote pull requests
| Code Block | ||
|---|---|---|
| ||
$ # Fetch remote pull requests
$ git fetch origin
$ # Checkout a remote pull request
$ git checkout origin/pr/112
$ # Create a local branch from a remote pull request
$ git checkout origin/pr/112 -b new-branch |
Running Individual Tests
Often it is useful to run individual tests in Maven or SBT.
| Code Block |
|---|
$ # sbt
$ sbt/sbt "test-only org.apache.spark.io.ComprsionCodecSuite"
$ sbt/sbt "test-only org.apache.spark.io.*"
$ # Maven
$ mvn clean test -DwildcardSuites=org.apache.spark.io.ComprsionCodecSuite
$ mvn clean test -DwildcardSuites=org.apache.spark.io.* |
Generating Dependency Graphs
| Code Block |
|---|
$ # sbt
$ sbt/sbt dependency-tree
$ # Maven
$ mvn -DskipTests install
$ mvn dependency:tree |
Running Build Targets For Individual Projects
...
Moved permanently to http://spark.apache.org/developer-tools.html