title: NoDefFound error
body: <p>I am getting the following errors while running Spark on Hadoop
  It is also giving me this error when I use the Scala API. I was convinced that the error is related to Spark paths and CLASSPATH.</p>
  
  <p>The error is:</p>
  
  <blockquote>
    <p>stage 0.0 failed 4 times; aborting job   17/04/25 13:36:53 WARN
    TaskSetManager: Lost task 11.2 in stage 0.0 (TID 20, 10.98.92.150,
    executor 4): TaskKilled (killed intentionally)    17/04/25 13:36:53 WARN
    TaskSetManager: Lost task 13.1 in stage 0.0 (TID 21, 10.98.92.150,
    executor 4): TaskKilled (killed intentionally)    17/04/25 13:36:53 WARN
    TaskSetManager: Lost task 6.0 in stage 0.0 (TID 6, 10.98.92.150,
    executor 5): TaskKilled (killed intentionally)    17/04/25 13:36:53 WARN
    TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, 10.98.92.150,
    executor 5): TaskKilled (killed intentionally)    Traceback (most recent
    call last):     File
    "/opt/apache/spark/spark-2.1.0-bin-hadoop2.7/examples/src/main/python/test.py",
    line 8, in 
          hive_context.sql("select count(1)  from src_tmp").show()      File "/opt/apache/spark/spark-2.1.0-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/sql/dataframe.py", line 318, in show       File
    "/opt/apache/spark/spark-2.1.0-bin-hadoop2.7/python/lib/py4j-0.10.4-src.zip/py4j/java_gateway.py",
    line 1133, in <strong>call</strong>      File
    "/opt/apache/spark/spark-2.1.0-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/sql/utils.py",
    line 63, in deco    File
    "/opt/apache/spark/spark-2.1.0-bin-hadoop2.7/python/lib/py4j-0.10.4-src.zip/py4j/protocol.py", line 319, in get_return_value  py4j.protocol.Py4JJavaError: An error
    occurred while calling o46.showString.    :
    org.apache.spark.SparkException: Job aborted due to stage failure:
    Task 5 in stage 0.0 failed 4 times, most recent failure: Lost task 5.3
    in stage 0.0 (TID 23, 10.98.92.151, executor 2):
    java.lang.NoClassDefFoundError: Could not initialize class
    org.apache.parquet.CorruptStatistics
              at org.apache.parquet.format.converter.ParquetMetadataConverter.fromParquetStatisticsInternal(ParquetMetadataConverter.java:346)
              at org.apache.parquet.format.converter.ParquetMetadataConverter.fromParquetStatistics(ParquetMetadataConverter.java:360)
              at org.apache.parquet.format.converter.ParquetMetadataConverter.fromParquetMetadata(ParquetMetadataConverter.java:816)
              at org.apache.parquet.format.converter.ParquetMetadataConverter.readParquetMetadata(ParquetMetadataConverter.java:793)
              at org.apache.parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:502)
              at org.apache.parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:461)
              at org.apache.parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:417)
              at org.apache.spark.sql.execution.datasources.parquet.SpecificParquetRecordReaderBase.initialize(SpecificParquetRecordReaderBase.java:107)
              at org.apache.spark.sql.execution.datasources.parquet.VectorizedParquetRecordReader.initialize(VectorizedParquetRecordReader.java:109)
              at org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat$$anonfun$buildReader$1.apply(ParquetFileFormat.scala:377)
              at org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat$$anonfun$buildReader$1.apply(ParquetFileFormat.scala:351)
              at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:150)
              at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:102)
              at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.scan_nextBatch$(Unknown
    Source)
              at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.agg_doAggregateWithoutKey$(Unknown
    Source)
              at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown
    Source)
              at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
              at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8$$anon$1.hasNext(WholeStageCodegenExec.scala:377)
              at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
              at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:126)
              at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
              at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
              at org.apache.spark.scheduler.Task.run(Task.scala:99)
              at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)        Driver stacktrace:
              at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1435)
              at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1423)
              at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1422)
              at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
              at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
              at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1422)
              at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:802)
              at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:802)
              at scala.Option.foreach(Option.scala:257)
              at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:802)
              at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1650)
              at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1605)
              at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1594)
              at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
              at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:628)
              at org.apache.spark.SparkContext.runJob(SparkContext.scala:1918)
              at org.apache.spark.SparkContext.runJob(SparkContext.scala:1931)
              at org.apache.spark.SparkContext.runJob(SparkContext.scala:1944)
              at org.apache.spark.sql.execution.SparkPlan.executeTake(SparkPlan.scala:333)
              at org.apache.spark.sql.execution.CollectLimitExec.executeCollect(limit.scala:38)
              at org.apache.spark.sql.Dataset$$anonfun$org$apache$spark$sql$Dataset$$execute$1$1.apply(Dataset.scala:2371)
              at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:57)
              at org.apache.spark.sql.Dataset.withNewExecutionId(Dataset.scala:2765)
              at org.apache.spark.sql.Dataset.org$apache$spark$sql$Dataset$$execute$1(Dataset.scala:2370)
              at org.apache.spark.sql.Dataset.org$apache$spark$sql$Dataset$$collect(Dataset.scala:2377)
              at org.apache.spark.sql.Dataset$$anonfun$head$1.apply(Dataset.scala:2113)
              at org.apache.spark.sql.Dataset$$anonfun$head$1.apply(Dataset.scala:2112)
              at org.apache.spark.sql.Dataset.withTypedCallback(Dataset.scala:2795)
              at org.apache.spark.sql.Dataset.head(Dataset.scala:2112)
              at org.apache.spark.sql.Dataset.take(Dataset.scala:2327)
              at org.apache.spark.sql.Dataset.showString(Dataset.scala:248)
              at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
              at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:498)
              at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
              at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
              at py4j.Gateway.invoke(Gateway.java:280)
              at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
              at py4j.commands.CallCommand.execute(CallCommand.java:79)
              at py4j.GatewayConnection.run(GatewayConnection.java:214)
              at java.lang.Thread.run(Thread.java:745)    Caused by: java.lang.NoClassDefFoundError: Could not initialize class
    org.apache.parquet.CorruptStatistics
              at org.apache.parquet.format.converter.ParquetMetadataConverter.fromParquetStatisticsInternal(ParquetMetadataConverter.java:346)
              at org.apache.parquet.format.converter.ParquetMetadataConverter.fromParquetStatistics(ParquetMetadataConverter.java:360)
              at org.apache.parquet.format.converter.ParquetMetadataConverter.fromParquetMetadata(ParquetMetadataConverter.java:816)
              at org.apache.parquet.format.converter.ParquetMetadataConverter.readParquetMetadata(ParquetMetadataConverter.java:793)
              at org.apache.parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:502)
              at org.apache.parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:461)
              at org.apache.parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:417)
              at org.apache.spark.sql.execution.datasources.parquet.SpecificParquetRecordReaderBase.initialize(SpecificParquetRecordReaderBase.java:107)
              at org.apache.spark.sql.execution.datasources.parquet.VectorizedParquetRecordReader.initialize(VectorizedParquetRecordReader.java:109)
              at org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat$$anonfun$buildReader$1.apply(ParquetFileFormat.scala:377)
              at org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat$$anonfun$buildReader$1.apply(ParquetFileFormat.scala:351)
              at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:150)
              at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:102)
              at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.scan_nextBatch$(Unknown
    Source)
              at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.agg_doAggregateWithoutKey$(Unknown
    Source)
              at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown
    Source)
              at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
              at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8$$anon$1.hasNext(WholeStageCodegenExec.scala:377)
              at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
              at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:126)
              at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
              at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
              at org.apache.spark.scheduler.Task.run(Task.scala:99)
              at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              ... 1 more</p>
  </blockquote>