title: Apply a custom function to a spark dataframe group
body: <p>I have a very big table of time series data that have these columns:</p>
  
  <ul>
  <li>Timestamp</li>
  <li>LicensePlate</li>
  <li>UberRide#</li>
  <li>Speed</li>
  </ul>
  
  <p>Each collection of LicensePlate/UberRide data should be processed considering the whole set of data. In others words, I <strong>do not</strong> need to proccess the data row by row, but all rows grouped by (LicensePlate/UberRide) together.</p>
  
  <p>I am planning to use spark with dataframe api, but I am confused on how can I perform a custom calculation over spark grouped dataframe.</p>
  
  <p>What I need to do is:</p>
  
  <ol>
  <li>Get all data</li>
  <li>Group by some columns</li>
  <li>Foreach spark dataframe group apply a f(x). Return a custom object foreach group</li>
  <li>Get the results by applying g(x) and returning a single custom object</li>
  </ol>
  
  <p>How can I do steps 3 and 4? Any hints over which spark API (dataframe, dataset, rdd, maybe pandas...) should I use?</p>
  
  <p>The whole workflow can be seen below:</p>
  
  <p><a href="https://i.stack.imgur.com/9XsB4.png" rel="noreferrer"><img src="https://i.stack.imgur.com/9XsB4.png" alt="Workflow"></a></p>