title: How to speed up recovery partition on S3 with Spark?
body: <p>I am using Spark 3.0 on EMR to write down some data on S3 with a daily partitioning (data goes back to ~5 years), in this way:</p>
  <pre><code>    writer.option(&quot;path&quot;, somepath).saveAsTable(&quot;my_schema.my_table&quot;)
  </code></pre>
  <p>Due to the large number of partitions the process is taking very long time just to &quot;recover partitions&quot; as all tasks seem completed before. Is there any way to reduce this intermediate time?</p>