Versions Compared

Old Version 21

changes.mady.by.user Reynold Xin

Saved on

compared with

New Version 22

changes.mady.by.user Madhu Siddalingaiah

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Add Eclipse configuration for Spark

...

To keep up to date with the latest discussions, join the developer mailing list.

IDE Setup

IntelliJ

While many of the Spark developers use SBT or Maven on the command line, the most common IDE we use is IntelliJ IDEA. You can get the community edition for free (Apache committers can get free IntelliJ Ultimate Edition licenses) and install the JetBrains Scala plugin from Preferences > Plugins. To generate an IDEA workspace for Spark, run

...

Then import the folder into IDEA. When you build the project, you might get a warning about "test and compile output paths" being the same for the "root-build" project. You can fix it by opening File -> Project Structure and changing the output path of the root-build module to be <spark-home>/project/target/idea-test-classes instead of idea-classes.

Eclipse

Eclipse can be used If you use Eclipse to develop and test Spark, feel free to add a short guide on setting it up to this wiki page.. The following configuration is known to work:

Scala IDE can be installed using Help | Eclipse Marketplace... and search for Scala IDE. Remember to include Scala Test as a Scala IDE plugin. To install Scala Test after installing Scala IDE, follow these steps:

  • Select Help | Install New Software
  • Select http://download.scala-ide.org... in the "Work with" combo box
  • Expand Scala IDE plugins, select ScalaTest for Scala IDE and install

SBT can create Eclipse .project and .classpath files. To create these files for each Spark sub project, use this command:

Code Block
sbt/sbt eclipse

To import a specific project, e.g. spark-core, select File | Import | Existing Projects into Workspace. Do not select "Copy projects into workspace". Importing all Spark sub projects at once is not recommended.

ScalaTest can execute unit tests by right clicking a source file and selecting Run As | Scala Test.

If Java memory errors occur, it might be necessary to increase the settings in eclipse.ini in the Eclipse install directory. Increase the following setting as needed:

Code Block
--launcher.XXMaxPermSize
256M