title: Spark + s3 - error - java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3a.S3AFileSystem not found
body: <p>I have a spark ec2 cluster where I am submitting a pyspark program from a Zeppelin notebook.  I have loaded the hadoop-aws-2.7.3.jar and aws-java-sdk-1.11.179.jar and place them in the /opt/spark/jars directory of the spark instances.  I get a java.lang.NoClassDefFoundError: com/amazonaws/AmazonServiceException</p>
  
  <p>Why is spark not seeing the jars?  Do I have to have to jars in all the slaves and specify a spark-defaults.conf for the master and slaves?  Is there something that needs to be configured in zeppelin to recognize the new jar files?</p>
  
  <p>I have placed jar files /opt/spark/jars on the spark master.  I have created a spark-defaults.conf and added the lines</p>
  
  <pre><code>spark.hadoop.fs.s3a.access.key     [ACCESS KEY]
  spark.hadoop.fs.s3a.secret.key     [SECRET KEY]
  spark.hadoop.fs.s3a.impl           org.apache.hadoop.fs.s3a.S3AFileSystem
  spark.driver.extraClassPath        /opt/spark/jars/hadoop-aws-2.7.3.jar:/opt/spark/jars/aws-java-sdk-1.11.179.jar
  </code></pre>
  
  <p>I have zeppelin interpreter sending a spark submit to the spark master.  </p>
  
  <p>I have also placed the jars in the /opt/spark/jars in the slaves too but did not create a spark-deafults.conf.</p>
  
  <pre><code>%spark.pyspark
  
  #importing necessary libaries
  from pyspark import SparkContext
  from pyspark.sql import SparkSession
  from pyspark.sql.functions import *
  from pyspark.sql.types import StringType
  from pyspark import SQLContext
  from itertools import islice
  from pyspark.sql.functions import col
  
  # add aws credentials
  sc._jsc.hadoopConfiguration().set("fs.s3n.awsAccessKeyId", "[ACCESS KEY]")
  sc._jsc.hadoopConfiguration().set("fs.s3n.awsSecretAccessKey", "[SECRET KEY]")
  sc._jsc.hadoopConfiguration().set("fs.s3a.impl", "org.apache.hadoop.fs.s3a.S3AFileSystem")
  
  #creating the context
  sqlContext = SQLContext(sc)
  
  #reading the first csv file and store it in an RDD
  rdd1= sc.textFile("s3a://filepath/baby-names.csv").map(lambda line: line.split(","))
  
  #removing the first row as it contains the header
  rdd1 = rdd1.mapPartitionsWithIndex(
  lambda idx, it: islice(it, 1, None) if idx == 0 else it
  )
  
  #converting the RDD into a dataframe
  df1 = rdd1.toDF(['year','name', 'percent', 'sex'])
  
  #print the dataframe
  df1.show()
  </code></pre>
  
  <p>Error thrown:</p>
  
  <pre><code>
  Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.runJob.
  : org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 1.0 failed 4 times, most recent failure: Lost task 0.3 in stage 1.0 (TID 7, 10.11.93.90, executor 1): java.lang.NoClassDefFoundError: com/amazonaws/AmazonServiceException
      at java.lang.Class.forName0(Native Method)
      at java.lang.Class.forName(Class.java:348)
      at org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:2134)
      at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2099)
      at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2193)
      at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2654)
      at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2667)
      at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)
      at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2703)
      at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2685)
      at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373)
      at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)
      at org.apache.hadoop.mapred.LineRecordReader.&lt;init&gt;(LineRecordReader.java:108)
      at org.apache.hadoop.mapred.TextInputFormat.getRecordReader(TextInputFormat.java:67)
      at org.apache.spark.rdd.HadoopRDD$$anon$1.liftedTree1$1(HadoopRDD.scala:267)
      at org.apache.spark.rdd.HadoopRDD$$anon$1.&lt;init&gt;(HadoopRDD.scala:266)
      at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:224)
      at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:95)
      at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
      at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
      at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
      at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
      at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
      at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:65)
      at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
      at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
      at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
      at org.apache.spark.scheduler.Task.run(Task.scala:123)
      at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
      at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
      at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      at java.lang.Thread.run(Thread.java:748)
  Caused by: java.lang.ClassNotFoundException: com.amazonaws.AmazonServiceException
      at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
      at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
      at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
      at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
      ... 34 more
  
  Driver stacktrace:
      at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1889)
      at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1877)
      at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1876)
      at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
      at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
      at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1876)
      at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:926)
      at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:926)
      at scala.Option.foreach(Option.scala:257)
      at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:926)
      at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2110)
      at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2059)
      at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2048)
      at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
      at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:737)
      at org.apache.spark.SparkContext.runJob(SparkContext.scala:2061)
      at org.apache.spark.SparkContext.runJob(SparkContext.scala:2082)
      at org.apache.spark.SparkContext.runJob(SparkContext.scala:2101)
      at org.apache.spark.api.python.PythonRDD$.runJob(PythonRDD.scala:153)
      at org.apache.spark.api.python.PythonRDD.runJob(PythonRDD.scala)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
      at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
      at py4j.Gateway.invoke(Gateway.java:282)
      at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
      at py4j.commands.CallCommand.execute(CallCommand.java:79)
      at py4j.GatewayConnection.run(GatewayConnection.java:238)
      at java.lang.Thread.run(Thread.java:748)
  Caused by: java.lang.NoClassDefFoundError: com/amazonaws/AmazonServiceException
      at java.lang.Class.forName0(Native Method)
      at java.lang.Class.forName(Class.java:348)
      at org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:2134)
      at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2099)
      at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2193)
      at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2654)
      at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2667)
      at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)
      at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2703)
      at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2685)
      at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373)
      at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)
      at org.apache.hadoop.mapred.LineRecordReader.&lt;init&gt;(LineRecordReader.java:108)
      at org.apache.hadoop.mapred.TextInputFormat.getRecordReader(TextInputFormat.java:67)
      at org.apache.spark.rdd.HadoopRDD$$anon$1.liftedTree1$1(HadoopRDD.scala:267)
      at org.apache.spark.rdd.HadoopRDD$$anon$1.&lt;init&gt;(HadoopRDD.scala:266)
      at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:224)
      at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:95)
      at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
      at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
      at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
      at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
      at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
      at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:65)
      at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
      at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
      at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
      at org.apache.spark.scheduler.Task.run(Task.scala:123)
      at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
      at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
      at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      ... 1 more
  Caused by: java.lang.ClassNotFoundException: com.amazonaws.AmazonServiceException
      at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
      at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
      at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
      at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
      ... 34 more
  </code></pre>