title: Spark + Amazon S3 "s3a://" urls
body: <p>AFAIK, the newest, best S3 implementation for Hadoop + Spark is invoked by using the "s3a://" url protocol. This works great on pre-configured Amazon EMR.</p>
  
  <p>However, when running on a local dev system using the pre-built <code>spark-2.0.0-bin-hadoop2.7.tgz</code>, I get</p>
  
  <pre><code>Caused by: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3a.S3AFileSystem not found
      at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2101)
      at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2193)
      ... 99 more
  </code></pre>
  
  <p>Next I tried to launch my Spark job specifying the hadoop-aws addon:</p>
  
  <pre><code>$SPARK_HOME/bin/spark-submit --master local \
      --packages org.apache.hadoop:hadoop-aws:2.7.3 \
      my_spark_program.py
  </code></pre>
  
  <p>I get</p>
  
  <pre><code>    ::::::::::::::::::::::::::::::::::::::::::::::
  
      ::              FAILED DOWNLOADS            ::
  
      :: ^ see resolution messages for details  ^ ::
  
      ::::::::::::::::::::::::::::::::::::::::::::::
  
      :: com.google.code.findbugs#jsr305;3.0.0!jsr305.jar
  
      :: org.apache.avro#avro;1.7.4!avro.jar
  
      :: org.xerial.snappy#snappy-java;1.0.4.1!snappy-java.jar(bundle)
  
      ::::::::::::::::::::::::::::::::::::::::::::::
  </code></pre>
  
  <p>I made a dummy build.sbt project in a temp directory with those three dependencies to see if a basic sbt build could successfully download those and I got:</p>
  
  <pre><code>[error] (*:update) sbt.ResolveException: unresolved dependency: org.apache.avro#avro;1.7.4: several problems occurred while resolving dependency: org.apache.avro#avro;1.7.4 {compile=[default(compile)]}:
  [error]     org.apache.avro#avro;1.7.4!avro.pom(pom.original) origin location must be absolute: file:/Users/username/.m2/repository/org/apache/avro/avro/1.7.4/avro-1.7.4.pom
  [error]     org.apache.avro#avro;1.7.4!avro.pom(pom.original) origin location must be absolute: file:/Users/username/.m2/repository/org/apache/avro/avro/1.7.4/avro-1.7.4.pom
  [error] 
  [error] unresolved dependency: com.google.code.findbugs#jsr305;3.0.0: several problems occurred while resolving dependency: com.google.code.findbugs#jsr305;3.0.0 {compile=[default(compile)]}:
  [error]     com.google.code.findbugs#jsr305;3.0.0!jsr305.pom(pom.original) origin location must be absolute: file:/Users/username/.m2/repository/com/google/code/findbugs/jsr305/3.0.0/jsr305-3.0.0.pom
  [error]     com.google.code.findbugs#jsr305;3.0.0!jsr305.pom(pom.original) origin location must be absolute: file:/Users/username/.m2/repository/com/google/code/findbugs/jsr305/3.0.0/jsr305-3.0.0.pom
  [error] 
  [error] unresolved dependency: org.xerial.snappy#snappy-java;1.0.4.1: several problems occurred while resolving dependency: org.xerial.snappy#snappy-java;1.0.4.1 {compile=[default(compile)]}:
  [error]     org.xerial.snappy#snappy-java;1.0.4.1!snappy-java.pom(pom.original) origin location must be absolute: file:/Users/username/.m2/repository/org/xerial/snappy/snappy-java/1.0.4.1/snappy-java-1.0.4.1.pom
  [error]     org.xerial.snappy#snappy-java;1.0.4.1!snappy-java.pom(pom.original) origin location must be absolute: file:/Users/username/.m2/repository/org/xerial/snappy/snappy-java/1.0.4.1/snappy-java-1.0.4.1.pom
  [error] Total time: 2 s, completed Sep 2, 2016 6:47:17 PM
  </code></pre>
  
  <p>Any ideas on how I can get this working?</p>