title: Spark DataFrame mapPartitions
body: <p>I need to proceed distributed calculation on Spark DataFrame invoking some arbitrary (not SQL) logic on chunks of DataFrame. 
  I did:</p>
  
  <pre><code>def some_func(df_chunk):
      pan_df = df_chunk.toPandas()
      #whatever logic here
  
  df = sqlContext.read.parquet(...)
  result = df.mapPartitions(some_func)
  </code></pre>
  
  <p>Unfortunatelly it leads to:</p>
  
  <blockquote>
    <p>AttributeError: 'itertools.chain' object has no attribute 'toPandas'</p>
  </blockquote>
  
  <p>I expected to have spark DataFrame object within each map invocation, instead I got 'itertools.chain'. Why? And how to overcome this?</p>