title: How to control partition size in Spark SQL
body: <p>I have a requirement to load data from an Hive table using Spark SQL <code>HiveContext</code> and load into HDFS. By default, the <code>DataFrame</code> from SQL output is having 2 partitions. To get more parallelism i need more partitions out of the SQL. There is no overloaded method in <code>HiveContex</code>t to take number of partitions parameter.</p>
  
  <p>Repartitioning of the RDD causes shuffling and results in more processing time.</p>
  
  <p>></p>
  
  <pre><code>val result = sqlContext.sql("select * from bt_st_ent")
  </code></pre>
  
  <p>Has the log output of:</p>
  
  <pre><code>Starting task 0.0 in stage 131.0 (TID 297, aster1.com, partition 0,NODE_LOCAL, 2203 bytes)
  Starting task 1.0 in stage 131.0 (TID 298, aster1.com, partition 1,NODE_LOCAL, 2204 bytes)
  </code></pre>
  
  <p>I would like to know is there any way to increase the partitions size of the SQL output.</p>