title: Parallelize pyspark 2.2.0 dataframe partitioned write to S3
body: <p>Starting to work with pyspark and run into a bottleneck I have created with my code:</p>
  
  <p>I'm "grouping by" pyspark 2.2.0 dataframe into partitions by drive_id
  and writing each partition (group) into its own location on S3.</p>
  
  <p>I need it to define Athena table on S3 location partitioned by drive_id - this allows me to read data very efficiently if queried by drive_id.</p>
  
  <pre><code>        #df is spark dataframe 
          g=df.groupBy(df.drive_id)
          rows=sorted(g.count().collect())
          #each row is a parition
          for row in rows:
              w=df.where((col("drive_id") == row.drive_id))
          w.write.mode('append').parquet("s3n://s3bucket/parquet/drives/"+str(table)+"/drive_id="+str(row.drive_id) )
  </code></pre>
  
  <p>The problem is that the loop makes processing serial and writes drive partitions only one by one.</p>
  
  <p>Obviously this doesn't scale well because single partition write task is quite small and parallelizing it doesn't give much.</p>
  
  <p>How do I replace the loop with single write command that will write all partitions into different locations ins a single operation? </p>
  
  <p>This operation should parallelize to run on spark workers, not driver.</p>