title: Spark coalesce vs collect, which one is faster?
body: <p>I am using <code>pyspark</code> to process 50Gb data using AWS EMR with ~15 m4.large cores. </p>
  
  <p>Each row of the data contains some information at a specific time on a day. I am using the following <code>for</code> loop to extract and aggregate information for every hour. Finally I <code>union</code> the data, as I want my result to save in <strong>one</strong> csv file.</p>
  
  <pre><code># daily_df is a empty pyspark DataFrame
  for hour in range(24):
      hourly_df = df.filter(hourFilter("Time")).groupby("Animal").agg(mean("weights"), sum("is_male"))
      daily_df = daily_df.union(hourly_df)
  </code></pre>
  
  <p>As of my knowledge, I have to perform the following to force the <code>pyspark.sql.Dataframe</code> object to save to 1 csv files (approx 1Mb) instead of 100+ files:</p>
  
  <pre><code>daily_df.coalesce(1).write.csv("some_local.csv")
  </code></pre>
  
  <p>It seems it took about 70min to finish this progress, and I am wondering if I can make it faster by using <code>collect()</code> method like?</p>
  
  <pre><code>daily_df_pandas = daily_df.collect()
  daily_df_pandas.to_csv("some_local.csv")
  </code></pre>