...
This works because PairFunction, DoubleFunction, and Function aren't subclasses of each other. Rather, this is the Function class hierarchy:
ClassManifests
...
AbstractFunction1 (scala.runtime)
WrappedFunction1 (org.apache.spark.api.java.function)
DoubleFunction (org.apache.spark.api.java.function)
PairFlatMapFunction (org.apache.spark.api.java.function)
PairFunction (org.apache.spark.api.java.function)
DoubleFlatMapFunction (org.apache.spark.api.java.function)
Function (org.apache.spark.api.java.function)
AbstractFunction2 (scala.runtime)
WrappedFunction2 (org.apache.spark.api.java.function)
Function2 (org.apache.spark.api.java.function)
|
ClassManifests
| Wiki Markup |
|---|
Many Spark methods take implicit [ClassManifest|http://www.scala-lang.org/api/2.9.3/scala/reflect/ClassManifest.html] arguments that are used by the compiler to preserve type information for instantiating Arrays at runtime. To hide ClassManifests from users, the Java API generates dummy ClassManifests by casting {{ClassManifest\[Object\]}} to the appropriate type. Users of the Java API have to work with RDDs of Java objects, since Java generics can't be parameterized with generic types, so this works fine. |
Methods with signatures like
| Code Block |
|---|
|
def transformationName[R: ClassManifest](args) |
are equivalent to
| Code Block |
|---|
|
def transformationName[R](args)(implicit cm: ClassManifest[R]) |
and will produce Java methods that accept explicit ClassManifest objects, making them difficult to call from Java:
| Code Block |
|---|
|
returnType transformationName(args, ClassManifest<? extends Object> evidence$1) |
| Wiki Markup |
|---|
Here's an excerpt from [JavaRDDLike.java|https://github.com/apache/incubator-spark/blob/0cef683553414ba880d90527cc5f37e119efc782/core/src/main/scala/org/apache/spark/api/java/JavaRDDLike.scala#L89], showing how we generate dummy manifests. In this example, the user-defined {{PairFunction}} has {{keyType()}} and {{valueType()}} methods to get its arguments' ClassManifests. To produce the {{Tuple2\[K2, V2\]}} ClassManifest, we just cast an Object class manifest: |
| Code Block |
|---|
|
/**
* Return a new RDD by applying a function to all elements of this RDD.
*/
def map[K2, V2](f: PairFunction[T, K2, V2]): JavaPairRDD[K2, V2] = {
def cm = implicitly[ClassManifest[AnyRef]].asInstanceOf[ClassManifest[Tuple2[K2, V2]]]
new JavaPairRDD(rdd.map(f)(cm))(f.keyType(), f.valueType())
}
|
Scala -> Java Types
TODO: complete
...