This page documents the design and internals of Spark's Java API and is intended for those developing Spark itself; if you are a user and want to learn to use Spark from Java, please see the Java programming guide.

This page is a draft; I'm still completing it. - Josh Rosen

Why a Java API?

Scala and Java are fairly interoperable, but there are several subtleties that make it difficult to directly call Spark's Scala APIs from Java:

• Spark uses Scala's implicit conversions to define additional operations on RDDs of key-value pairs and doubles, such as `reduceByKey`, `join`, and `stdev`.
  Since Java doesn't support implicit conversions, users have to manually instantiate the `PairRDDFunctions` and `DoubleRDDFunctions` classes to access these methods.
• Many Spark functions take implicit `ClassManifest` arguments; users have to manually pass `ClassManifest` instances when calling these functions from Java.
• To express user-defined functions in Java, users have to subclass Scala's internal function classes, which can be confusing.
• Many of Spark's methods accept or return Scala collection types; this is inconvenient and often results in users manually converting to and from Java types.

These difficulties made for an unpleasant user experience. To address this, the Spark 0.7 release introduced a Java API that hides these Scala <-> Java interoperability concerns.

Implementation

Function Classes

Scala -> Java Types

Workarounds for compiler bugs

SI-6050 / SI-3452: problems using implementation traits

Many RDD transformations obey a "same-result-type" principle where they derive RDDs of the same type. For example, applying `filter` to a JavaPairRDD should yield a new JavaPairRDD, while applying `filter` to JavaDoubleRDD yields a JavaDoubleRDD.

Many operations in the Scala collections library obey this same principle. To avoid code duplication, the Scala collections define implementation traits, like TraversableLike, that contain methods that are parameterized by the implementor's type. This allows them to define methods like `filter` with types like "function from a predicate defined on elements of this collection to a new concrete collection of the same type."

Ideally, we could use this same technique to factor out all of the common operations in JavaRDD, JavaPairRDD, and JavaDoubleRDD into the JavaRDDLike class. Unfortunately, this doesn't seem to work properly due to Scala compiler bugs; in our case, the compiler produced code that compiled fine but threw NoSuchMethodErrors at runtime (SI-6050 is a minimal example that reproduces this problem).

A Scala pull request proposed a workaround for this bug. When compiling the code with the '-Ycheck:genjvm' flag, as suggested in that pull request, the compiler emits warnings when compiling the Scala code and compile-time typechecking errors when calling that Scala code from Java. This is an improvement, since it makes the problem explicit during compilation, but it doesn't fix the problem.

To work around this issue, methods that must return RDDs of the same type (like filter and distinct) are redundantly implemented in each Java*RDD class. As a result, these methods aren't defined in JavaRDDLike. This limits users' abilities to define functions that accept JavaRDDLike and perform transformations on them, since transformations like filter and distinct won't be available via that interface.

When these compiler bugs are fixed, we should refactor the code to remove these redundancies and define a proper implementation trait.

SI-6057: NoSuchMethodErrors due to type parameter naming

TODO: this section doesn't precisely state the conditions for this problem to occur.

In some rare cases, type parameters in different scopes can interfere in a way that causes problems when applying methods. For example, see SI-6057.

This affected a few methods in the Java API, including JavaRDDLike.map(PairFunction).

Keeping the Java API up-to-date

TODO: Describe JavaAPICompletenessChecker