title: Spark Sql sort operation is not working properly
body: <p>I am trying to sort a dataframe with <strong>sort</strong> function but it dosen't sort properly and seems it is sorting in chunks. Maybe it is sorting the individual partitions and not combing the sort.However where clause is working fine </p>
  
  <p>I have used sort function and plain sql query also , none of them seems to work</p>
  
  <pre><code>System.setProperty("hadoop.home.dir", "C:\\winutils");    
  val tempDir = "file:///c:/temp/spark-warehouse"    
  Logger.getLogger("org").setLevel(Level.ERROR)
  // Create a SparkContext using every core of the local machine, named RatingsCounter
  
  val sparkSession = SparkSession
    .builder()
    .appName("AppStore")
    .master("local[2]")
    .config("spark.sql.warehouse.dir", tempDir)
    .getOrCreate()
  
  var appStoreDF = sparkSession.read.format("csv").option("header", "true").load("../AppleStore.csv") 
  appStoreDF.show(10)
  import org.apache.spark.sql.functions._
  
  appStoreDF.select("size_bytes").sort(desc("size_bytes")).show(10)  // Gives unsorted results
  </code></pre>
  
  <p>this is the output I get:</p>
  
  <pre><code>+----------+
  |size_bytes|
  +----------+
  |  99992576|
  |  99978240|
  | 999398400|
  |  99890176|
  |  99851264|
  |  99829760|
  |  99796992|
  | 997171200|
  | 997129216|
  |  99691520|
  +----------+
  </code></pre>