title: SparkSQL DataFrame order by across partitions
body: <p>I'm using spark sql to run a query over my dataset. The result of the query is pretty small but still partitioned.</p>
  
  <p>I would like to coalesce the resulting DataFrame and order the rows by a column. I tried</p>
  
  <pre><code>DataFrame result = sparkSQLContext.sql("my sql").coalesce(1).orderBy("col1")
  result.toJSON().saveAsTextFile("output")
  </code></pre>
  
  <p>I also tried</p>
  
  <pre><code>DataFrame result = sparkSQLContext.sql("my sql").repartition(1).orderBy("col1")
  result.toJSON().saveAsTextFile("output")
  </code></pre>
  
  <p>the output file is ordered in chunks (i.e. the partitions are ordered, but the data frame is not ordered as a whole). For example, instead of</p>
  
  <pre><code>1, value
  2, value
  4, value
  4, value
  5, value
  5, value
  ...
  </code></pre>
  
  <p>I get</p>
  
  <pre><code>2, value
  4, value
  5, value
  -----------&gt; partition boundary
  1, value
  4, value
  5, value
  </code></pre>
  
  <ol>
  <li>What is the correct way to get an absolute ordering of my query result?</li>
  <li>Why isn't the data frame being coalesced into a single partition?</li>
  </ol>