title: pySpark convert result of mapPartitions to spark DataFrame
body: <p>I have a job requires to run on a partitioned spark dataframe, and the process looks like:</p>
  
  <pre><code>rdd = sp_df.repartition(n_partitions, partition_key).rdd.mapPartitions(lambda x: some_function(x))
  </code></pre>
  
  <p>The result is an <code>rdd</code> of <code>pandas.dataframe</code>, </p>
  
  <pre><code>type(rdd) =&gt; pyspark.rdd.PipelinedRDD
  type(rdd.collect()[0]) =&gt; pandas.core.frame.DataFrame
  </code></pre>
  
  <p>and <code>rdd.glom().collect()</code> returns result like:</p>
  
  <pre><code>[[df1], [df2], ...]
  </code></pre>
  
  <p>Now I hope to convert the result to a spark dataframe, the way I did is:</p>
  
  <pre><code>sp = None
  for i, partition in enumerate(rdd.collect()):
      if i == 0:
          sp = spark.createDataFrame(partition)
      else:
          sp = sp.union(spark.createDataFrame(partition))
  
  return sp
  
  </code></pre>
  
  <p>However, the result could be huge and <code>rdd.collect()</code> may exceed driver's memory, so I need to avoid <code>collect()</code> operation. Is there a way to address the problem?</p>
  
  <p>Thanks in advance!</p>