title: Optimization of Bucket and Sort operation
body: <pre><code># Code snippet 
  # sparkBucketSize = 2000
  # partition column based on hash of uniqueId into 100
    dataWithPartitionDF.repartition(sparkBucketSize).write
                  .option(&quot;compression&quot;, &quot;snappy&quot;)
                  .option(&quot;path&quot;,partition_bucket_s3 + &quot;/bucketed/&quot; + day['date'])\
                  .mode(&quot;overwrite&quot;)\
                  .format(&quot;parquet&quot;)\
                  .partitionBy(&quot;partition&quot;)\
                  .bucketBy(sparkBucketSize, &quot;uniqueId&quot;)\
                  .sortBy(&quot;uniqueId&quot;)\
                  .saveAsTable(&quot;bucketed&quot;);
  </code></pre>
  <p>Investigations</p>
  <ul>
  <li>History Server. There are 2000 task and it takes (~2.8 hrs) for 240.9 MB input with 315066 records per task. Tasks are mapped to each core. Ideally it should not take this much time to sort records. I need help in bucket and sort operation here. Are they correctly written. Are there any hidden gotchas in Sort after BucketBy operation.</li>
  </ul>