title: Spark/ Scala- Faster Way to Load Dataframe in Hive Table
body: <p>I have a data frame which I intend to write as a Hive partitioned table. The code I use to that end is</p>
  
  <pre><code>for(i &lt;- 0 until 10)
  {
    myDf = hiveContext.sql("select * from srcTable where col = "+i)
    myDf.write.mode("append").format("parquet")
          .partitionBy("period_id").saveAsTable("myTable")
  }
  </code></pre>
  
  <p><code>myDf</code> will contain a different set of data in every iteration (I have just shown an oversimplified way of how I get values in <code>myDf</code>)</p>
  
  <p>The <code>myDf.write</code> takes about 5 minutes to load 120,000 rows of data. Is there any way I could further reduce the time taken to write all this data?</p>