title: Best way to send Spark dataframe as JSON body over HTTP POST
body: <p>I have a <code>Spark dataframe</code> that I need to send as body of <code>HTTP POST</code> request. The storage system is <code>Apache Solr</code>. We are creating <code>Spark dataframe</code> by reading <code>Solr</code> collection. I can use <code>Jackson</code> library to create <code>JSON</code> and send it over <code>HTTP POST</code>. Also, dataframe may have millions of records so preferred way is to send them in <code>batches</code> over HTTP.</p>
  
  <p>Below are the two approaches I can think.</p>
  
  <ul>
  <li><p>We can use <code>foreach</code>/<code>foreachPartition</code> operations of <code>Spark dataframe</code> and call <code>HTTP POST</code> which means that HTTP call will happen within each executor (If I am not wrong). Is this approach right? Also, it means if I have 3 <code>executors</code> then there will be 3 HTTP calls that we can make in parallel. Right? But opening and closing HTTP connection so many times, will it not cause issue?</p></li>
  <li><p>After getting the <code>Spark dataframe</code>, we can save it in some other <code>SOLR</code> collection (using Spark) and then data from that collection will be read to get the data in batches using <code>SOLR API</code> (using <code>rows, start</code> parameters), create JSON out of it and send it over HTTP request.</p></li>
  </ul>
  
  <p>I would like to know which one of the above two approaches is preferred?</p>