title: PySpark Access DataFrame columns at foreachPartition() custom function
body: <p>I have a function named "inside". I want to apply this function to a pyspark dataframe. For this purpose i call the "foreachPartition(inside)" method on the dataframe I create. The "inside" function needs the values of the dataframe. </p>
  
  <p>The dataframe looks like this:</p>
  
  <pre><code>&gt;&gt;&gt; small_df
  DataFrame[lon: double, lat: double, t: bigint]
  </code></pre>
  
  <p>The code looks like this:</p>
  
  <pre><code>def inside(iterator):
      row=iterator
      x=row.lon
      y=row.lat
      i=row.t 
      #do more stuff
  
  small=pliades.iloc[0:20000,:] #take sample of rows from big dataset
  small_df=sqlContext.createDataFrame(small) #create dataframe
  test=small_df.foreachPartition(inside)
  </code></pre>
  
  <p>My question is: how can x,y,i get the values of the first(lon),second(lat) and third(t) columns of the dataframe respectively?</p>
  
  <p>I tried also doing it with row.lon, row.select, treating it as a list but couldn't get the result needed.</p>