title: Pyspark job being stuck at the final task
body: <p>The flow of my program is something like this: <br/><br/>
  1. Read 4 billion rows (~700GB) of data from a parquet file into a data frame. Partition size used is 2296 <br/> <br/> 
  2. Clean it and filter out 2.5 billion rows <br/> <br/>
  3. Transform the remaining 1.5 billion rows using a pipeline model and then a trained model. The model is trained using a logistic regression model where it predicts 0 or 1 and 30% of the data is filtered out of the transformed data frame. <br/> <br/>
  4. The above data frame is Left outer joined with another dataset of ~1 TB (also read from a parquet file.) Partition size is 4000 <br/> <br/>
  5. Join it with another dataset of around 100 MB like <br/> <br/>
  <code>joined_data = data1.join(broadcast(small_dataset_100MB), data1.field == small_dataset_100MB.field, "left_outer")</code> <br/> <br/>
  6. The above dataframe is then exploded to the factor of ~2000 <br/><br/><code>exploded_data = joined_data.withColumn('field', explode('field_list'))</code> <br/> <br/>
  7. An aggregate is performed <br/><br/><code>aggregate = exploded_data.groupBy(*cols_to_select)\
          .agg(F.countDistinct(exploded_data.field1).alias('distincts'), F.count("*").alias('count_all'))</code> <br/> <br/> There are a total of 10 columns in the <code>cols_to_select</code> list. <br/> <br/>
  8. And finally an action, <code>aggregate.count()</code> is performed.</p>
  
  <p>The problem is, the third last count stage (200 tasks) gets stuck at task 199 forever. In spite of allocating 4 cores and 56 executors, the count uses only one core and one executor to run the job. I tried breaking down the size from 4 billion rows to 700 million rows which is 1/6th part, it took four hours. I would really appreciate some help in how to speed this process up Thanks</p>