title: Making HTTP post requests on Spark usign foreachPartition
body: <p>Need some help to understand the behaviour of the below in Spark (using Scala and Databricks)</p>
  
  <p>I have some dataframe (reading from S3 if that matters), and would send that data by making HTTP post requests in batches of 1000 (at most). So I repartitioned the dataframe to make sure each partition has no more than 1000 records. Also, created a json column for each line (so I need only to put them in an array later on)</p>
  
  <p>The trouble is on the making the requests. I created the following a Serializable class using the following code</p>
  
  <pre><code>import org.apache.spark.sql.{DataFrame, Row}
  import org.apache.http.client.methods.HttpPost
  import org.apache.http.impl.client.HttpClientBuilder
  import org.apache.http.HttpHeaders
  import org.apache.http.entity.StringEntity
  import org.apache.commons.io.IOUtils
  
  object postObject extends Serializable{
    val client = HttpClientBuilder.create().build()
    val post = new HttpPost("https://my-cool-api-endpoint")
    post.addHeader(HttpHeaders.CONTENT_TYPE,"application/json")
    def makeHttpCall(row: Iterator[Row]) = {
        val json_str = """{"people": [""" + row.toSeq.map(x =&gt; x.getAs[String]("json")).mkString(",") + "]}"      
        post.setEntity(new StringEntity(json_str))
        val response = client.execute(post)
        val entity = response.getEntity()
        println(Seq(response.getStatusLine.getStatusCode(), response.getStatusLine.getReasonPhrase()))
        println(IOUtils.toString(entity.getContent()))
    }
  }
  </code></pre>
  
  <p>Now when I try the following:</p>
  
  <pre><code>postObject.makeHttpCall(data.head(2).toIterator)
  </code></pre>
  
  <p>It works like a charm. The requests go through, there is some output on the screen, and my API gets that data.</p>
  
  <p>But when I try to put it in the foreachPartition:</p>
  
  <pre><code>data.foreachPartition { x =&gt; 
    postObject.makeHttpCall(x)
  }
  </code></pre>
  
  <p>Nothing happens. No output on screen, nothing arrives in my API. If I try to rerun it, almost all stages just skips. I believe, for any reason, it is just lazy evaluating my requests, but not actually performing it. I don't understand why, and how to force it.</p>