title: Reading CSV using spark-csv package in spark-shell
body: <p>I am trying to use <a href="https://github.com/databricks/spark-csv" rel="nofollow">spark-csv</a> to read a csv from aws s3 in spark-shell.</p>
  
  <p>Below are the steps that I did. Started spark-shell using below command</p>
  
  <blockquote>
    <p>bin/spark-shell --packages com.databricks:spark-csv_2.10:1.2.0</p>
  </blockquote>
  
  <p>In the shell, executed the following scala code</p>
  
  <pre><code>scala&gt; val hadoopConf = sc.hadoopConfiguration
  scala&gt; hadoopConf.set("fs.s3.impl", "org.apache.hadoop.fs.s3native.NativeS3FileSystem")
  scala&gt; hadoopConf.set("fs.s3.awsAccessKeyId", "****")
  scala&gt; hadoopConf.set("fs.s3.awsSecretAccessKey", "****")
  
  scala&gt; val s3path = "s3n://bucket/sample.csv"
  scala&gt; val df = sqlContext.read.format("com.databricks.spark.csv").option("header", "true").load(s3path)
  </code></pre>
  
  <p>Getting the below error</p>
  
  <pre><code>java.io.IOException: No FileSystem for scheme: s3n
      at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2584)
      at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591)
      at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
      at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)
      at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)
      at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
      at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
      at org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:256)
      at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:228)
      at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:313)
      at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:207)
      at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:219)
      at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:217)
      at scala.Option.getOrElse(Option.scala:120)
  </code></pre>
  
  <p>What is that I am missing here? Please note that I am able to read the csv using</p>
  
  <pre><code>scala&gt; sc.textFile(s3path)
  </code></pre>
  
  <p>The same scala code is working fine in <a href="https://databricks.com/product/databricks" rel="nofollow">databricks notebook</a> as well</p>
  
  <p>Created a <a href="https://github.com/databricks/spark-csv/issues/137" rel="nofollow">issue</a> in spark-csv github. I'll update here when I get answer for the <a href="https://github.com/databricks/spark-csv/issues/137" rel="nofollow">issue</a> </p>