• Created by Unknown User (joshrosen), last updated on Oct 23, 2013 3 minute read

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 3 Next »

This page documents the design and internals of Spark's Java API and is intended for those developing Spark itself; if you are a user and want to learn to use Spark from Java, please see the Java programming guide.

Why a Java API?

Scala and Java are fairly interoperable, but there are several subtleties that make it difficult to directly call Spark's Scala APIs from Java:

  • Spark uses Scala's implicit conversions to define additional operations on RDDs of key-value pairs and doubles, such as `reduceByKey`, `join`, and `stdev`.
    Since Java doesn't support implicit conversions, users have to manually instantiate the `PairRDDFunctions` and `DoubleRDDFunctions` classes to access these methods.
  • Many Spark functions take implicit `ClassManifest` arguments; users have to manually pass `ClassManifest` instances when calling these functions from Java.
  • To express user-defined functions in Java, users have to subclass Scala's internal function classes, which can be confusing.
  • Many of Spark's methods accept or return Scala collection types; this is inconvenient and often results in users manually converting to and from Java types.

These difficulties made for an unpleasant user experience. To address this, the Spark 0.7 release introduced a Java API that hides these Scala <-> Java interoperability concerns.

Implementation

The Java API is implemented as a thin wrapper over the equivalent Scala APIs. The bulk of this wrapper is actually implemented using Scala, except we use Java-friendly types and hide Scala-specific features, like ClassManifest.

Spark defines additional operations on RDDs of key-value pairs and doubles, such as reduceByKey, join, and stdev. In the Scala API, these methods are automatically added using Scala’s implicit conversions mechanism. In the Java API, the extra methods are defined in the JavaPairRDD and JavaDoubleRDD classes.

Function Classes

Transformations like `map` return different RDDs depending on the type of function that's applied to the RDD's elements. Due to type erasure, this overloading is ambiguous and won't work:

def map[R](f: Function1[T, R]): JavaRDD[R]
def map(f: Function1[T, Double]): JavaDoubleRDD
def map[K, V](f: Function1[T, (K, V)]): JavaPairRDD[K, V]

Instead, we define a hierarchy of Java Function classes that allow functions like map to be properly overloaded:

def map[R](f: Function[T, R]): JavaRDD[R]
def map[R](f: DoubleFunction[T]): JavaDoubleRDD
def map[K2, V2](f: PairFunction[T, K2, V2]): JavaPairRDD[K2, V2]

This works because PairFunction, DoubleFunction, and Function aren't subclasses of each other. Rather, this is the Function class hierarchy:

TODO

ClassManifests

TODO: complete

Scala -> Java Types

TODO: complete

Workarounds for compiler bugs

SI-6050 / SI-3452: problems using implementation traits

Many RDD transformations obey a "same-result-type" principle where they derive RDDs of the same type. For example, applying `filter` to a JavaPairRDD should yield a new JavaPairRDD, while applying `filter` to JavaDoubleRDD yields a JavaDoubleRDD.

Many operations in the Scala collections library obey this same principle. To avoid code duplication, the Scala collections define implementation traits, like TraversableLike, that contain methods that are parameterized by the implementor's type. This allows them to define methods like `filter` with types like "function from a predicate defined on elements of this collection to a new concrete collection of the same type."

Ideally, we could use this same technique to factor out all of the common operations in JavaRDD, JavaPairRDD, and JavaDoubleRDD into the JavaRDDLike class. Unfortunately, this doesn't seem to work properly due to Scala compiler bugs; in our case, the compiler produced code that compiled fine but threw NoSuchMethodErrors at runtime (SI-6050 is a minimal example that reproduces this problem).

A Scala pull request proposed a workaround for this bug. When compiling the code with the '-Ycheck:genjvm' flag, as suggested in that pull request, the compiler emits warnings when compiling the Scala code and compile-time typechecking errors when calling that Scala code from Java. This is an improvement, since it makes the problem explicit during compilation, but it doesn't fix the problem.

To work around this issue, methods that must return RDDs of the same type (like filter and distinct) are redundantly implemented in each Java*RDD class. As a result, these methods aren't defined in JavaRDDLike. This limits users' abilities to define functions that accept JavaRDDLike and perform transformations on them, since transformations like filter and distinct won't be available via that interface.

When these compiler bugs are fixed, we should refactor the code to remove these redundancies and define a proper implementation trait.

SI-6057: NoSuchMethodErrors due to type parameter naming

TODO: this section doesn't precisely state the conditions for this problem to occur.

In some rare cases, type parameters in different scopes can interfere in a way that causes problems when applying methods. For example, see SI-6057.

This affected a few methods in the Java API, including JavaRDDLike.map(PairFunction).

Keeping the Java API up-to-date

TODO: Describe JavaAPICompletenessChecker

  • No labels