title: Spark Dataset Parquet partition on s3 creating temporary folder
body: <p>Spark<code>(version=2.2.0)</code> there is not <code>DirectParquetOutputCommitter</code>. As an alternative, I can use</p>
  
  <pre><code>dataset
      .option("mapreduce.fileoutputcommitter.algorithm.version", "2")//magic here
      .parquet("s3a://...")
  </code></pre>
  
  <p>to avoid creating <code>_temporary</code> folder on <code>S3</code>. </p>
  
  <p>Everything works fine until I set a <code>partitionBy</code> to my Dataset</p>
  
  <pre><code>dataset
      .partitionBy("a", "b")
      .option("mapreduce.fileoutputcommitter.algorithm.version", "2")//magic stop working creating _temporary on S3
      .parquet("s3a://...")
  </code></pre>
  
  <p>Also tried adding but didn't work</p>
  
  <pre><code>spark.conf.set("mapreduce.fileoutputcommitter.algorithm.version", "2")
  </code></pre>
  
  <p>With <code>partitionBy</code> at Spark Dataset, It's going to create <code>_temporary</code> and move files which becomes a very slow operation.</p>
  
  <p>There is any alternative or missing configuration?</p>