title: Overwriting a file in Azure datalake Gen 2 from Synapse Notebook throws Exception
body: <p>As part of migrating from Azure Databricks to Azure Synapse Analytics Notebooks, I'm facing the issue explained below.</p>
  <p>While reading a CSV file from Azure Datalake Storage Gen 2 and assigning it to a pyspark dataframe using the following command.</p>
  <pre><code>df = spark.read.format('csv').option(&quot;delimiter&quot;, &quot;,&quot;).option(&quot;multiline&quot;, &quot;true&quot;).option(&quot;quote&quot;, '&quot;').option(&quot;header&quot;, &quot;true&quot;).option(&quot;escape&quot;, &quot;\\&quot;).load(csvFilePath)
  </code></pre>
  <p>After processing this file, we need to overwrite it and we use the following command.</p>
  <pre><code>df.coalesce(1).write.option(&quot;delimiter&quot;, &quot;,&quot;).csv(csvFilePath, mode = 'overwrite', header = 'true')
  </code></pre>
  <p>What this does is, it deletes the existing file at the path &quot;csvFilePath&quot; and the fails with error, &quot;Py4JJavaError: An error occurred while calling o617.csv.&quot;</p>
  <p>Things I've noticed:</p>
  <ol>
  <li>Once the CSV file at path &quot;csvFilePath&quot; is deleted by the overwrite command, data from dataframe &quot;df&quot; also gets removed.</li>
  <li>Looks like it is referring the file at runtime whereas traditionally in databricks we did not have this issue and overwrite ran successfully.</li>
  </ol>
  <p>[Error returned by Synapse Notebook at write command.][1]
  [1]: https://i.stack.imgur.com/Obj9q.png</p>