title: Last 2/3 task takes huge time compared to all other task in spark
body: <p>I am trying to do sentimental analysis of comments. Program is running successfully on Spark, But the problem I am facing is out of 70 partitions 68 partitions gave result in around 20% of time compared to the time taken by last 2 partitions. I have checked my data is equally distributed on all partitions , and even checked with different sample data also. </p>
  
  <p>Also I had run the code using <code>persist(StorageLevel.MEMORY_AND_DISK_SER)</code> for all dataframes, and unpersist these dataframes as soon as they are not required anymore.</p>
  
  <p>I also tried increasing and decreasing the number of partitions, but still for last 2 tasks, It takes huge time. Following is the current config I am using</p>
  
  <pre><code>--master yarn \
  --deploy-mode client \
  --num-executors 15 \
  --executor-cores 5 \
  --executor-memory 32g \
  --driver-memory 8g \
  --driver-cores 8 \
  </code></pre>
  
  <p>I have used KryoSerializer. Following is set in sparkConf</p>
  
  <pre><code>sparkConf.set("spark.driver.allowMultipleContexts", "true")
  sparkConf.set("spark.scheduler.mode", "FAIR")
  sparkConf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
  sparkConf.set("spark.kryoserializer.buffer.max", "1024m")
  </code></pre>
  
  <p>How can I optimise, so that last 2 partitions dont take so much time?
  Thanks</p>