title: pySpark forEachPartition - Where is code executed
body: <p>I'm using pySpark in version 2.3 (cannot update to 2.4 in my current dev-System) and have the following questions concerning the <a href="http://spark.apache.org/docs/latest/api/python/pyspark.html#pyspark.RDD.foreachPartition" rel="nofollow noreferrer">foreachPartition</a>.</p>
  <p>First a little context: As far as I understood pySpark-<code>UDFs</code> force the Python-code to be executed outside the Java Virtual Machine (JVM) in a Python-instance, making it performance-costing.
  Since I need to apply some Python-functions to my data and want to minimize overhead costs, I had the idea to at least load a handable bunch of data into the driver and process it as Pandas-DataFrame. Anyhow, this would lead to a loss of the parallelism-advantage Spark has.
  Then I read that <code>foreachPartition</code> applies a function to all the data within a partition and, hence, allows parallel processing.</p>
  <p>My questions now are:</p>
  <ol>
  <li><p>When I apply a Python-function via <code>foreachPartition</code>, does the Python-execution take place within the driver process (and the partition-data is therefore transfered over the network to my driver)?</p>
  </li>
  <li><p>Is the data processed row-wise within <code>foreachPartition</code> (meaning every RDD-row is transfered one by one to the Python-instance), or is the partition-data processed at once (meaning, for example, the whole partition is transfered to the instance and is handled as whole by one Python-instance)?</p>
  </li>
  </ol>
  <p>Thank you in advance for your input!</p>
  <hr />
  <p><em><strong>Edit:</strong></em></p>
  <p>A working in driver-solution I used before looks like this, taken from SO <a href="https://stackoverflow.com/a/48995752/3757672">here</a>:</p>
  <pre class="lang-py prettyprint-override"><code>for partition in rdd.mapPartitions(lambda partition: [list(partition)]).toLocalIterator():
      # Do stuff on the partition
  </code></pre>
  <p>As can be read from the <a href="https://spark.apache.org/docs/2.4.0/api/python/pyspark.html#pyspark.RDD.toLocalIterator" rel="nofollow noreferrer">docs</a> <code>rdd.toLocalIterator()</code> provides the necessary functionality:</p>
  <blockquote>
  <p>Return an iterator that contains all of the elements in this RDD. The iterator <strong>will consume as much memory as the largest partition</strong> in this RDD.</p>
  </blockquote>