title: How to sort array of struct type in Spark DataFrame by particular field?
body: <p>Given following code:</p>
  <pre><code>import java.sql.Date
  import org.apache.spark.sql.SparkSession
  import org.apache.spark.sql.functions._
  
  object SortQuestion extends App{
  
    val spark = SparkSession.builder().appName(&quot;local&quot;).master(&quot;local[*]&quot;).getOrCreate()
    import spark.implicits._
    case class ABC(a: Int, b: Int, c: Int)
  
    val first = Seq(
      ABC(1, 2, 3),
      ABC(1, 3, 4),
      ABC(2, 4, 5),
      ABC(2, 5, 6)
    ).toDF(&quot;a&quot;, &quot;b&quot;, &quot;c&quot;)
  
    val second = Seq(
      (1, 2, (Date.valueOf(&quot;2018-01-02&quot;), 30)),
      (1, 3, (Date.valueOf(&quot;2018-01-01&quot;), 20)),
      (2, 4, (Date.valueOf(&quot;2018-01-02&quot;), 50)),
      (2, 5, (Date.valueOf(&quot;2018-01-01&quot;), 60))
    ).toDF(&quot;a&quot;, &quot;b&quot;, &quot;c&quot;)
  
    first.join(second.withColumnRenamed(&quot;c&quot;, &quot;c2&quot;), Seq(&quot;a&quot;, &quot;b&quot;)).groupBy(&quot;a&quot;).agg(sort_array(collect_list(&quot;c2&quot;)))
      .show(false)
  
  }
  </code></pre>
  <p>Spark produces following result:</p>
  <pre class="lang-none prettyprint-override"><code>+---+----------------------------------+
  |a  |sort_array(collect_list(c2), true)|
  +---+----------------------------------+
  |1  |[[2018-01-01,20], [2018-01-02,30]]|
  |2  |[[2018-01-01,60], [2018-01-02,50]]|
  +---+----------------------------------+
  </code></pre>
  <p>This implies that Spark is sorting an array by date (since it is the first field), but I want to instruct Spark to sort by specific field from that nested struct.</p>
  <p>I know I can reshape array to <code>(value, date)</code> but it seems inconvenient, I want a general solution (imagine I have a big nested struct, 5 layers deep, and I want to sort that structure by particular column).</p>
  <p>Is there a way to do that? Am I missing something?</p>