NOTE: This Wiki is obsolete as of November 2016 and is retained for reference only.

This page documents the design and internals of Spark's Java API and is intended for those developing Spark itself; if you are a user and want to learn to use Spark from Java, please see the Java programming guide.

...

Scala and Java are fairly interoperable, but there are several subtleties that make it difficult to directly call Spark's Scala APIs from Java:

• Spark uses Scala's implicit conversions to define additional operations on RDDs of key-value pairs and doubles, such as `reduceByKey`, `join`, and `stdev`.
  Since Java doesn't support implicit conversions, users have to manually instantiate the `PairRDDFunctions` and `DoubleRDDFunctions` classes to access these methods.
• Many Spark functions take implicit `ClassManifest` arguments; users have to manually pass `ClassManifest` instances when calling these functions from Java.
• To express user-defined functions in Java, users have to subclass Scala's internal function classes, which can be confusing.
• Many of Spark's methods accept or return Scala collection types; this is inconvenient and often results in users manually converting to and from Java types.

These difficulties made for an unpleasant user experience. To address this, the Spark 0.7 release introduced a Java API that hides these Scala <-> Java interoperability concerns.

...

Transformations like `map` return different RDDs depending on the type of function that's applied to the RDD's elements. Due to type erasure, this overloading is ambiguous and won't work:

def map[R](f: Function1[T, R]): JavaRDD[R]
def map(f: Function1[T, Double]): JavaDoubleRDD
def map[K, V](f: Function1[T, (K, V)]): JavaPairRDD[K, V]

Instead, we define a hierarchy of Java Function classes that allow functions like map to be properly overloaded:

def map[R](f: Function[T, R]): JavaRDD[R]
def map[R](f: DoubleFunction[T]): JavaDoubleRDD
def map[K2, V2](f: PairFunction[T, K2, V2]): JavaPairRDD[K2, V2]

This works because PairFunction, DoubleFunction, and Function aren't subclasses of each other. Rather, this is the Function class hierarchy:

AbstractFunction1 (scala.runtime)
    WrappedFunction1 (org.apache.spark.api.java.function)
        DoubleFunction (org.apache.spark.api.java.function)
        PairFlatMapFunction (org.apache.spark.api.java.function)
        PairFunction (org.apache.spark.api.java.function)
        DoubleFlatMapFunction (org.apache.spark.api.java.function)
        Function (org.apache.spark.api.java.function)

AbstractFunction2 (scala.runtime)
    WrappedFunction2 (org.apache.spark.api.java.function)
        Function2 (org.apache.spark.api.java.function)

ClassManifests

Wiki MarkupMany Spark methods take implicit [ClassManifest|http://www.scala-lang.org/api/2.9.3/scala/reflect/ClassManifest.html] arguments that are used by the compiler to preserve type information for instantiating Arrays at runtime. To hide ClassManifests from users, the Java API generates dummy ClassManifests by casting {{ClassManifest\[Object\]}} to the appropriate type. Users of the Java API have to work with RDDs of Java objects, since Java generics can't be parameterized with generic types, so this works fine.

Methods with signatures like

...

returnType transformationName(args, ClassManifest<? extends Object> evidence$1)

...

Here's an excerpt from [JavaRDDLike.java|https://github.com/apache/incubator-spark/blob/0cef683553414ba880d90527cc5f37e119efc782/core/src/main/scala/org/apache/spark/api/java/JavaRDDLike.scala#L89], showing how we generate dummy manifests. In this example, the user-defined {{PairFunction}} has {{keyType()}} and {{valueType()}} methods to get its arguments' ClassManifests. To produce the {{Tuple2\[K2, V2\]}} ClassManifest, we just cast an Object class manifest:

  /**
   * Return a new RDD by applying a function to all elements of this RDD.
   */
  def map[K2, V2](f: PairFunction[T, K2, V2]): JavaPairRDD[K2, V2] = {
    def cm = implicitly[ClassManifest[AnyRef]].asInstanceOf[ClassManifest[Tuple2[K2, V2]]]
    new JavaPairRDD(rdd.map(f)(cm))(f.keyType(), f.valueType())
  }

...

The Java API exposes standard Java collections types, instead of Scala ones. These substitutions include:

• scala.collection.Map -> java.util.Map
• scala.collection.Seq -> java.util.List (or in some cases, array)
• scala.collection.Iterator -> java.util.Iterator
• scala.collection.mutable.Queue -> java.util.Queue
• scala.Option -> com.google.common.base.Optional

Most of these conversions are performed using Scala's JavaConverters package.

...

Guidelines for adding methods to the Java API

• No user-facing ClassManifests: Java API methods should never contain ClassManifest in their signatures, including type bounds.
• Methods added to JavaRDD should also be added to Java(Double|Pair)RDD: All of JavaRDD's methods obey the "same-result-type" principle (if they didn't, they should have been implemented in JavaRDDLike), so they should also be implemented in JavaDoubleRDD and JavaPairRDD.
• Handling default parameters: Java methods don't support optional parameters with default values. For Scala methods with optional arguments, like def textFile(path: String, minSplits: Int = defaultMinSplits), you can either define two separate methods (1- and 2-parameter versions) or make the optional argument into a required one.
• Unit tests: for non-trivial additions that might be prone to compile-time or run-time problems if untested, please add new unit tests to JavaAPISuite.

References and Resources

• Twitter's Scala School "Java + Scala" guide provides a nice overview of Java interoperability issues and explores some of the details of how Scala constructs are exposed in Java.