title: Spark: java.io.FileNotFoundException: File does not exist in copyMerge
body: <p>I am trying to merge all spark output part files in a directory and create a single file in Scala.</p>
  
  <p>Here is my code:</p>
  
  <pre><code>import org.apache.spark.sql.functions.input_file_name
  import org.apache.spark.sql.functions.regexp_extract
  
  def merge(srcPath: String, dstPath: String): Unit =  {
     val hadoopConfig = new Configuration()
     val hdfs = FileSystem.get(hadoopConfig)
     FileUtil.copyMerge(hdfs, new Path(srcPath), hdfs, new Path(dstPath), true, hadoopConfig, null) 
     // the "true" setting deletes the source files once they are merged into the new output
  }
  </code></pre>
  
  <p>And then at last step, I am writing data frame output like below.</p>
  
  <pre><code>dfMainOutputFinalWithoutNull.repartition(10).write.partitionBy("DataPartition","StatementTypeCode")
    .format("csv")
    .option("nullValue", "")
    .option("header", "true")
    .option("codec", "gzip")
    .mode("overwrite")
    .save(outputfile)
    merge(mergeFindGlob, mergedFileName )
    dfMainOutputFinalWithoutNull.unpersist()
  </code></pre>
  
  <p>When I run this I get below exception </p>
  
  <pre><code>java.io.FileNotFoundException: File does not exist: hdfs:/user/zeppelin/FinancialLineItem/temp_FinancialLineItem
    at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1309)
  </code></pre>
  
  <p>This is how I get my output </p>
  
  <p><a href="https://i.stack.imgur.com/7Be1Y.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/7Be1Y.png" alt="enter image description here"></a></p>
  
  <p>Instead of the folder, I want to merge all files inside a folder and create a single file. </p>