title: Pyspark - Load file: Path does not exist
body: <p>I am a newbie to Spark. I'm trying to read a local csv file within an EMR cluster. The file is located in: /home/hadoop/. The script that I'm using is this one:</p>
  
  <pre><code>spark = SparkSession \
      .builder \
      .appName("Protob Conversion to Parquet") \
      .config("spark.some.config.option", "some-value") \
      .getOrCreate()\
  
  df = spark.read.csv('/home/hadoop/observations_temp.csv, header=True)
  </code></pre>
  
  <p>When I run the script raises the following error message:</p>
  
  <blockquote>
    <p>pyspark.sql.utils.AnalysisException: u'Path does not exist:
    hdfs://ip-172-31-39-54.eu-west-1.compute.internal:8020/home/hadoop/observations_temp.csv</p>
  </blockquote>
  
  <p>Then, I found out that I have to add file:// in the file path so it can read the file locally:</p>
  
  <pre><code>df = spark.read.csv('file:///home/hadoop/observations_temp.csv, header=True)
  </code></pre>
  
  <p>But this time, the above approach raised a different error:</p>
  
  <blockquote>
    <p>Lost task 0.3 in stage 0.0 (TID 3,<br>
    ip-172-31-41-81.eu-west-1.compute.internal, executor 1):
    java.io.FileNotFoundException: File
    file:/home/hadoop/observations_temp.csv does not exist</p>
  </blockquote>
  
  <p>I think is because the file// extension just read the file locally and it does not distribute the file across the other nodes.</p>
  
  <p>Do you know how can I read the csv file and make it available to all the other nodes?</p>