Versions Compared

Old Version 41

changes.mady.by.user Patrick Wendell

Saved on

compared with

New Version 42

changes.mady.by.user Sean Owen

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Updated IntelliJ notes per discussion at https://github.com/apache/spark/pull/3952

...

  1. Download IntelliJ and install the Scala plug-in for IntelliJ.
  2. Go to "File -> Import Project", locate the spark source directory, and select "Maven Project".
  3. Click through to the profiles selection, and select the following profiles: yarn, scala-2.10, hadoop-2.4, hive-thriftserver, hive-0.13.1. Click through to create the project.
  4. At the top of the leftmost pane, make sure the "Project/Packages" selector is set to "Packages".
  5. Right click on any package and click “Open Module Settings” - you will be able to modify any of the modules here.
    6. A few of the modules need to be modified slightly from the default import.
    1. Add sources to the following modules: Under “Sources” tab add a source on the right. 
      1. spark-hive: add v0.13.1/src/main/scala
      2. spark-hive-thriftserver v0.13.1/src/main/scala
      3. spark-repl: scala-2.10/src/main/scala and scala-2.10/src/test/scala
    For spark-yarn click “Add content root” and navigate in the filesystem to yarn/common directory of Spark
  6. In the Import wizard, it's fine to leave settings at their default. However it is usually useful to enable "Import Maven projects automatically", since changes to the project structure will automatically update the IntelliJ project.
  7. As documented in Building Spark, some build configurations require specific profiles to be enabled. The same profiles that are enabled with -P[profile name] above may be enabled on the Profiles screen in the Import wizard. For example, if developing for Hadoop 2.4 with YARN support, enable profiles yarn and hadoop-2.4. These selections can be changed later by accessing the "Maven Projects" tool window from the View menu, and expanding the Profiles section.

Other tips:

  • "Rebuild Project" can fail the first time the project is compiled, because generate source files are not automatically generated. Try clicking the "Generate Sources and Update Folders For All Projects" button in the "Maven Projects" tool window to manually generate these sources.
  • Compilation may fail with an error like "scalac: bad option: -P:/home/jakub/.m2/repository/org/scalamacros/paradise_2.10.4/2.0.1/paradise_2.10.4-2.0.1.jar". If so, go to Preferences > Build, Execution, Deployment > Scala Compiler and clear the "Additional compiler options" field. It will work then although the option will come back when the project reimports.

Eclipse

Eclipse can be used to develop and test Spark. The following configuration is known to work:

...