title: How to convert Pandas Dataframe coming from RDD.mapPartitions() into Spark DataFrame?
body: <p>I've got a Python function that returns a Pandas DataFrame. I'm calling this function in Spark 2.2.0 using pyspark's <a href="https://spark.apache.org/docs/2.2.0/api/python/pyspark.html#pyspark.RDD.mapPartitions" rel="nofollow noreferrer"><code>RDD.mapPartitions()</code></a>. But I can't convert the RDD returned by <code>mapPartitions()</code> into a Spark DataFrame. Pandas generates this error:</p>
  
  <pre><code>ValueError: The truth value of a DataFrame is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().
  </code></pre>
  
  <p>Simple code which illustrates the problem:</p>
  
  <pre><code>import pandas as pd
  
  def func(data):
      pdf = pd.DataFrame(list(data), columns=("A", "B", "C"))
      pdf += 10 # Add 10 to every value. The real function is a lot more complex!
      return [pdf]
  
  pdf = pd.DataFrame([(1.87, 0.6, 7.1), (-0.3, 0.1, 8.2), (2.8, 0.3, 6.1), (-0.2, 0.5, 5.9)], columns=("A", "B", "C"))
  
  sdf = spark.createDataFrame(pdf)
  sdf.show()
  rddIn = sdf.rdd
  
  for i in rddIn.collect():
      print(i)
  
  result = rddIn.mapPartitions(func)
  
  for i in result.collect():
      print(i)
  
  resDf = spark.createDataFrame(result) # --&gt; ValueError!
  resDf.show()
  </code></pre>
  
  <p>The output is:</p>
  
  <pre><code>+----+---+---+
  |   A|  B|  C|
  +----+---+---+
  |1.87|0.6|7.1|
  |-0.3|0.1|8.2|
  | 2.8|0.3|6.1|
  |-0.2|0.5|5.9|
  +----+---+---+
  Row(A=1.87, B=0.6, C=7.1)
  Row(A=-0.3, B=0.1, C=8.2)
  Row(A=2.8, B=0.3, C=6.1)
  Row(A=-0.2, B=0.5, C=5.9)
         A     B     C
  0  11.87  10.6  17.1
       A     B     C
  0  9.7  10.1  18.2
        A     B     C
  0  12.8  10.3  16.1
       A     B     C
  0  9.8  10.5  15.9
  </code></pre>
  
  <p>but the second to last line produces the <code>ValueError</code> mentioned above. I really want <code>resDf.show()</code> to look exactly the same as <code>sdf.show()</code> except with 10 added to every value in the table. Ideally the <code>result</code> RDD should have the same structure as <code>rddIn</code>, the RDD going in to <code>mapPartitions()</code>.</p>