title: Using s3n:// with spark-submit
body: <p>I've written a Spark app that is to run on a cluster using <code>spark-submit</code>. Here's part of my <code>build.sbt</code>.</p>
  
  <pre><code>libraryDependencies += "org.apache.spark" %% "spark-core" % "1.6.1" % "provided" exclude("asm", "asm")
  libraryDependencies += "org.apache.spark" %% "spark-sql" % "1.6.1" % "provided"
  </code></pre>
  
  <p><code>asm</code> is excluded because I'm using another library which depends on a different version of it. The <code>asm</code> dependency in Spark seems to come from one of Hadoop's dependents, and I'm not using the functionality.</p>
  
  <p>The problem now is that with this setup, <code>saveToTextFile("s3n://my-bucket/dir/file")</code> throws <code>java.io.IOException: No FileSystem for scheme: s3n</code>.</p>
  
  <p>Why is this happening? Shouldn't <code>spark-submit</code> provide the Hadoop dependencies?</p>
  
  <p>I've tried a few things:</p>
  
  <ul>
  <li>Leaving out <code>"provided"</code></li>
  <li>Putting <code>hadoop-aws</code> on the classpath, via a jar and <code>spark.executor.extraClassPath</code> and <code>spark.driver.extraClassPath</code>. This requires doing the same for all of its transitive dependencies though, which can be painful.</li>
  </ul>
  
  <p>Neither really works. Is there a better approach?</p>
  
  <p>I'm using the pre-built <code>spark-1.6.1-bin-hadoop2.6</code>.</p>