title: PySpark / Spark Window Function First/ Last Issue
body: <p>From my understanding first/ last function in Spark will retrieve first / last row of each partition/ I am not able to understand why <strong>LAST</strong> function is giving incorrect results.</p>
  
  <p>This is my code.</p>
  
  <pre class="lang-py prettyprint-override"><code>AgeWindow = Window.partitionBy('Dept').orderBy('Age')
  df1 = df1.withColumn('first(ID)', first('ID').over(AgeWindow))\
          .withColumn('last(ID)', last('ID').over(AgeWindow))           
  df1.show()
  </code></pre>
  
  <pre class="lang-none prettyprint-override"><code>+---+----------+---+--------+--------------------------+-------------------------+
  |Age|      Dept| ID|    Name|first(ID)                 |last(ID)                |
  +---+----------+---+--------+--------------------------+-------------------------+
  | 38|  medicine|  4|   harry|                         4|                        4|
  | 41|  medicine|  5|hermione|                         4|                        5|
  | 55|  medicine|  7| gandalf|                         4|                        7|
  | 15|technology|  6|  sirius|                         6|                        6|
  | 49|technology|  9|     sam|                         6|                        9|
  | 88|technology|  1|     sam|                         6|                        2|
  | 88|technology|  2|     nik|                         6|                        2|
  | 75|       mba|  8|   ginny|                         8|                       11|
  | 75|       mba| 10|     sam|                         8|                       11|
  | 75|       mba|  3|     ron|                         8|                       11|
  | 75|       mba| 11|     ron|                         8|                       11|
  +---+----------+---+--------+--------------------------+-------------------------+
  </code></pre>