Versions Compared

Old Version 21

changes.mady.by.user Brock Noland

Saved on

compared with

New Version 22

changes.mady.by.user Brock Noland

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

IssueCauseResolution

java.lang.NoSuchMethodError: com.google.common.hash.HashFunction.hashInt

(I)Lcom/google/common/hash/HashCode

Guava library version conflict between Spark and Hadoop.  See HIVE-7387 and SPARK-2420 for details.

Alternatives until this is fixed:

  1. Remove guava 11 from HADOOP_HOME and replace it with guava 14
  2. Choose to build Spark assembly manually, apply HIVE-7387-spark.patch to Spark branch before building.

org.apache.spark.SparkException: Job aborted due to stage failure:

Task 5.0:0 had a not serializable result: java.io.NotSerializableException: org.apache.hadoop.io.BytesWritable

Spark serializer not set to KryoSet spark.serializer to be org.apache.spark.serializer.KryoSerializer as described above

java.lang.NullPointerException

at org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:257)

at org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:224)

Hive is included in the Spark AssemblyEither build a version of spark without the "hive" profile or unjar the spark assembly and rm -rf org/apache/hive org/apache/hadoop/hive and then rejar. The fix is in SPARK-2741