title: How handle uneven partition in spark
body: <p>So i have a data frame which output 300 GB records into S3 .
  All the data is partitioned in 2K partition. </p>
  
  <pre><code>.partitionBy("DataPartition", "PartitionYear", "PartitionStatement")
  </code></pre>
  
  <p>But the issue is some of the partition has very huge data (40GB) and some has only 10MB .</p>
  
  <p>So if i re partition it again like <code>.repartition(100)</code> then it creates many files for even 10MB data size partition also and that leads to huge no of output files .
  When i try to load all the output files and run my spark job then because of many small files my spark job becomes very slow .</p>
  
  <p>What i am looking for is can we re partition only those data frames which has huge no of records .</p>
  
  <p>Lest say i know for below partition<code>(Japan 1970 BAL)</code> the records will be huge so can we repartition that partition only ?</p>
  
  <p>I did not found anything about this on internet .</p>
  
  <p>I tried even this </p>
  
  <pre><code>dfMainOutputFinalWithoutNull.repartition($"DataPartition", $"PartitionYear", $"PartitionStatement")
        .write
        .partitionBy("DataPartition", "PartitionYear")
  </code></pre>
  
  <p>But this also did not turns out in good performance .</p>
  
  <p>Please suggest something so that my output files no also not become every huge and my job also runs faster .</p>
  
  <p>This is not working for me </p>
  
  <pre><code> val rddFirst = sc.textFile(mainFileURL)
      val rdd = rddFirst.repartition(190)
  </code></pre>
  
  <p>Because when i try to split the file name i get error .</p>