title: Overwrite a Parquet file with Pyspark
body: <p>I am trying to overwrite a Parquet file in S3 with Pyspark.  Versioning is enabled for the bucket.</p>
  <p>I am using the following code:</p>
  <p>Write v1:</p>
  <pre><code>df_v1.repartition(1).write.parquet(path='s3a://bucket/file1.parquet')
  </code></pre>
  <p>Update v2:</p>
  <pre><code>df_v1 = spark.read.parquet(&quot;s3a://bucket/file1.parquet&quot;)
  df_v2 = df_v1.... &lt;- transform
  df_v2.repartition(1).write.mode(&quot;overwrite&quot;).parquet('s3a://bucket/file1.parquet')
  </code></pre>
  <p>But when I read df_v2 it contains data from both writes.  Furthermore when df_v1 is written I can see one part-xxx.snappy.parquet file, after writing df_v2 I can see two.  It behaves as an append rather than overwrite.</p>
  <p>What am I missing ? Thanks</p>
  <p>Spark = 2.4.4
  Hadoop = 2.7.3</p>