title: PYSPARK - Solution for slow performance when writing dataframe to parquet file when using repartition() before partitionby()?
body: <p>I want to write my data (contained in a dataframe) into parquet files.
  I need to partition the data by two variables : "month" and "level". (data is always filtered on these 2 variables)</p>
  
  <p>If I do the following 
  <code>data.write.format("parquet").partitionBy("month", "level").save("...")</code> I end up with the expected partitions, however i have a lot of files per partitions. Some of these files are really small which hurt the performance of queries run on the data.</p>
  
  <p>In order to correct that, I tried to apply <code>repartition</code> before writing the data :
  <code>data.repartition("month", "level").write.format("parquet").partitionBy("month", "level").save("...")</code> which give me exactly what i want (1 file per partition, with a decent size for each file).</p>
  
  <p>===> the problem here is that the repartition causes a full shuffle of the data, which means that for an input data of 400Gb, I end up with a few Tb of shuffle...</p>
  
  <p>Is there any way to optimize the repartition() before the partitionby() or to do this any other way ?</p>
  
  <p>Thanks !</p>