title: Databricks: dbutils.fs.mv() throws java.io.FileNotFoundException although file exists
body: <p>I have a function to replace parquet files in ADLS Gen2:</p>
  <pre class="lang-py prettyprint-override"><code>def replace_parquet_file(df: DataFrame, path: str):
    path_new = path + '_new'
    path_old = path + '_old'
    
    if not file_exists(path):
      df.write.mode('overwrite').parquet(path)
    else:  
      df.write.parquet(path_new)
  
      dbutils.fs.mv(path, path_old, recurse = True)
      dbutils.fs.mv(path_new, path, recurse = True)
      dbutils.fs.rm(path_old, recurse = True)
  </code></pre>
  <p>This function keeps failing at the second mv() operation with the following error:</p>
  <pre><code> 135 
      136     dbutils.fs.mv(path, path_old, recurse = True)
  --&gt; 137     dbutils.fs.mv(path_new, path, recurse = True)
      138     dbutils.fs.rm(path_old, recurse = True)
      139 
  
  java.io.FileNotFoundException: Failed with java.io.FileNotFoundException while processing file/directory :[/tmp/myfile.parquet/_committed_6614329046368392922] in method:[Operation failed: &amp;#34;The specified path does not exist.&amp;#34;, 404, PUT, https://XXX.dfs.core.windows.net/XXX/tmp/myfile.parquet/_committed_6614329046368392922?action=append&amp;amp;position=0&amp;amp;timeout=90, PathNotFound, &amp;#34;The specified path does not exist. ...] 
  </code></pre>
  <p>This is obviously a partition, written by spark. As this is the second mv() operation, my assumption would be spark has already finished writing all partitions. When I look at the ADLS myself, the file clearly exists.</p>
  <p>I do not understand, what happens here? Is the write process by spark not finished at the moment the mv() operation ist started or is this more of an error with the ADLS API?</p>
  <p>Best,
  Jan</p>