title: Getting last value of group in Spark
body: <p>I have a SparkR DataFrame as shown below:</p>
  
  <pre><code>#Create R data.frame
  custId &lt;- c(rep(1001, 5), rep(1002, 3), 1003)
  date &lt;- c('2013-08-01','2014-01-01','2014-02-01','2014-03-01','2014-04-01','2014-02-01','2014-03-01','2014-04-01','2014-04-01')
  desc &lt;- c('New','New','Good','New', 'Bad','New','Good','Good','New')
  newcust &lt;- c(1,1,0,1,0,1,0,0,1)
  df &lt;- data.frame(custId, date, desc, newcust)
  
  #Create SparkR DataFrame    
  df &lt;- createDataFrame(df)
  display(df)
        custId|    date   | desc | newcust
        --------------------------------------
         1001 | 2013-08-01| New  |   1
         1001 | 2014-01-01| New  |   1
         1001 | 2014-02-01| Good |   0
         1001 | 2014-03-01| New  |   1
         1001 | 2014-04-01| Bad  |   0
         1002 | 2014-02-01| New  |   1
         1002 | 2014-03-01| Good |   0
         1002 | 2014-04-01| Good |   0 
         1003 | 2014-04-01| New  |   1
  </code></pre>
  
  <p><code>newcust</code> indicates a new customer every time a new <code>custId</code> appears, or if the same <code>custId</code>'s <code>desc</code> reverts to 'New'. What I want to obtain is the last <code>desc</code> value for each grouping of <code>newcust</code>, while maintaining the first <code>date</code> for each grouping. Below is the DataFrame I want to obtain. How can I do this in Spark? Either PySpark or SparkR code will work.</p>
  
  <pre><code>#What I want 
  custId|    date   | newcust | finaldesc
  ----------------------------------------------
   1001 | 2013-08-01|   1     | New
   1001 | 2014-01-01|   1     | Good
   1001 | 2014-03-01|   1     | Bad
   1002 | 2014-02-01|   1     | Good
   1003 | 2014-04-01|   1     | New
  </code></pre>