title: Spark Structured Streaming : mapPartitions is supported?
body: <p>Is it possible to use mapPartitions in Spark Structured streaming ?</p>
  
  <p>Running into these errors </p>
  
  <p>option 1: </p>
  
  <pre class="lang-py prettyprint-override"><code>dataframe_python.mapPartitions(processfunction)
  </code></pre>
  
  <p>'DataFrame' object has no attribute 'mapPartitions'</p>
  
  <p>option 2:</p>
  
  <pre class="lang-py prettyprint-override"><code>dataframe_python.rdd.mapPartitions(processfunction);
  </code></pre>
  
  <p>'Queries with streaming sources must be executed with writeStream.start();</p>
  
  <p>is there a way to use mapPartitions for my scenario ? my intention is to transform the existing dataframe to another dataframe while minimizing the calls to external resource API by sending batch of rows.</p>
  
  <p>For example processfunction will look like below : </p>
  
  <pre class="lang-py prettyprint-override"><code>processfunction(rows):
  
   batch = list(rows)
   results = call_external_resource(batch)
   for row in rows 
    tmp_row = rows[i]
    tmp_row["new_column"] = results[i]
    yield Row(**tmp_row)
  </code></pre>