title: java.io.FileNotFoundException: File file:/data/home/test/analysis.json does not exist. Spark error while reading json
body: <p>I am trying to write a json file from a spark/scala program and then read it into a DataFrame. This is my code:</p>
  
  <pre><code> val analysisWriter = new BufferedWriter(new FileWriter("analysis.json"))
   for(i &lt;- 0 to 10){
           val obj =  arr.get(i).asInstanceOf[JSONObject]
           currentAnalysis(""+obj.get("id"))
      }
      analysisWriter.close()
      val df = hiveContext.read.json("file:///data/home/test/analysis.json")
      df.show(10)
  
    }   
  
    def currentAnalysis(id: String): Unit= {
       val arrCurrentAnalysis: JSONObject = acc.getCurrentAnalysis(""+id)
  
       if(arrCurrentAnalysis != null) {
         analysisWriter.append(arrCurrentAnalysis.toString())
         analysisWriter.newLine()
    }
  </code></pre>
  
  <p>I get the following error when I try to run this code:</p>
  
  <pre><code>java.io.FileNotFoundException: File file:/data/home/test/analysis.json does not exist
  </code></pre>
  
  <p>I can see the file being created in the same directory where the jar(I am running the jar using <code>spark-submit</code>) is present. Why is the code not able to find the file?</p>
  
  <p>Initially, I was getting <code>java.io.IOException: No input paths specified in job</code></p>
  
  <p>As pointed out here : <a href="https://stackoverflow.com/questions/38150311/spark-sql-no-input-paths-specified-in-jobs-when-create-dataframe-based-on-json">Spark SQL &quot;No input paths specified in jobs&quot; when create DataFrame based on JSON file</a></p>
  
  <p>and here: <a href="https://stackoverflow.com/questions/43800187/spark-java-io-ioexception-no-input-paths-specified-in-job">Spark java.io.IOException: No input paths specified in job</a> ,</p>
  
  <p>I added <code>file://</code> to the path to read the json file from and now I get the <code>FileNotFoundException</code>. </p>
  
  <p>I am running <code>spark 1.6</code> on a <code>yarn</code> cluster. Could it be the case that the file is not being available to the executors as it was created after the program has been launched?</p>