title: Read file on remote machine in Apache Spark using ftp
body: <p>I am trying to read a file on an remote machine in Apache Spark (the Scala version) using ftp. Currently, I have followed an example in the Learning Spark repo of Databricks on GitHub. Using curl, I am able to download the file, so the path I uses exists. </p>
  
  <p>Below is a snippet of the code I try to execute:</p>
  
  <pre><code>var file = sc.textFile("ftp://user:pwd/192.168.1.5/brecht-d-m/map/input.nt")
  var fileDF = file.toDF()
  fileDF.write.parquet("out")
  </code></pre>
  
  <p>After trying to execute a count on the dataframe, I get following stacktrace (<a href="http://pastebin.com/YEq8c2Hf" rel="noreferrer">http://pastebin.com/YEq8c2Hf</a>):</p>
  
  <pre><code>org.apache.spark.sql.catalyst.errors.package$TreeNodeException: execute, tree:
  TungstenAggregate(key=[], functions=[(count(1),mode=Final,isDistinct=false)], output=[count#1L])
  +- TungstenExchange SinglePartition, None
     +- TungstenAggregate(key=[], functions=[(count(1),mode=Partial,isDistinct=false)], output=[count#4L])
        +- Project
           +- Scan ExistingRDD[_1#0]
  
  ...
  
  Caused by: org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: ftp://user:pwd@192.168.1.5/brecht-d-m/map/input.nt
  </code></pre>
  
  <p>I would assume that the file would be unreachable, but this is in contradiction with that I am able to retrieve the file via curl:</p>
  
  <pre><code>curl ftp://user:pwd@192.168.1.5/brecht-d-m/map/input.nt
  </code></pre>
  
  <p>This will print out the specific file on my terminal. I do not see what I am doing wrong in the Scala code. Is there an error in the code snippet I gave above, or is that code totally wrong? </p>
  
  <p>Thanks in advance, 
  Brecht</p>
  
  <p>Note: </p>
  
  <ul>
  <li><p>Specifying the whole path (/home/brecht-d-m/map/input.nt) also does not work (as expected, since this also does not work in curl; "server denied you to change to the given directory"). Trying this in Spark, gives the IOException that seek is not supported (<a href="http://pastebin.com/b9EB9ru2" rel="noreferrer">http://pastebin.com/b9EB9ru2</a>). </p></li>
  <li><p>Working locally (e.g. sc.textFile("/home/brecht-d-m/map/input.nt")) works perfectly.</p></li>
  <li><p>File permissions for specific file is set to R+W for all users.</p></li>
  <li><p>The file size (15MB) should not be a problem, and it should be able to handle much bigger files.</p></li>
  <li><p>Software versions: Scala 2.11.7, Apache Spark 1.6.0, Java 1.8.0_74, Ubuntu 14.04.4</p></li>
  </ul>