title: Can UDFs be forced to execute in row order (by group) in spark?
body: <p>I want to apply arbitrary scala window-aware aggregating functions to a Dataset. e.g. multiply the last 3 rows of &quot;column&quot; (silly, but simple example).</p>
  <p>Many of these could be converted to native spark functions, but not all (external libraries etc) - and I don't want to have to, since they are used as-is in other applications (and simply want replication in spark).
  Each of these functions folds/aggregates the new row and returns a new value (for that row, discarding any rows out of range).</p>
  <p>How can I effectively call these arbitrary functions in row order by group? (obviously within a partition - I don't think it can work across partitions).</p>
  <p>For example:</p>
  <pre><code>trait UpdatingFunction {
    /* adds new value, discards oldest value and calculates result */
    def newValue(i: Int): Int
  }
  
  class AddValue2RowsPrior extends UpdatingFunction {...}
  
  df.select(col(&quot;value&quot;), udf(new AddValue2RowsPrior)(col(&quot;value&quot;))).show()
  </code></pre>
  <p>Would produce something like</p>
  <div class="s-table-container">
  <table class="s-table">
  <thead>
  <tr>
  <th>value</th>
  <th>AddValue2RowsPrior</th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td>1</td>
  <td>1</td>
  </tr>
  <tr>
  <td>2</td>
  <td>2</td>
  </tr>
  <tr>
  <td>3</td>
  <td>4 //(3+1)</td>
  </tr>
  <tr>
  <td>4</td>
  <td>6 //(4+2)</td>
  </tr>
  <tr>
  <td>5</td>
  <td>8</td>
  </tr>
  </tbody>
  </table>
  </div>
  <p>My current ideas are (commentary welcome):</p>
  <ul>
  <li>A dynamic class-based UDF with groups and ordering might just work without anything extra</li>
  <li>Convert <code>value</code> column(s) to arrays and process as a (very large) row</li>
  <li>UDAF with a array/dataset output value?</li>
  <li><code>RDD.map</code></li>
  <li>Use windows and just use the functions as UDAFs that operate on every window, effectively discarding the folding mechanism. (Appealing, since a day of the computer's time is cheaper than my time - but oh, so annoying)</li>
  </ul>