title: User defined function to be applied to Window in PySpark?
body: <p>I am trying to apply a user defined function to Window in PySpark. I have read that UDAF might be the way to to go, but I was not able to find anything concrete.</p>
  
  <p>To give an example (taken from here: <a href="http://xinhstechblog.blogspot.com/2016/04/spark-window-functions-for-dataframes.html" rel="noreferrer">Xinh's Tech Blog</a> and modified for PySpark):</p>
  
  <pre class="lang-py prettyprint-override"><code>from pyspark import SparkConf
  from pyspark.sql import SparkSession
  from pyspark.sql.window import Window
  from pyspark.sql.functions import avg
  
  spark = SparkSession.builder.master("local").config(conf=SparkConf()).getOrCreate()
  
  a = spark.createDataFrame([[1, "a"], [2, "b"], [3, "c"], [4, "d"], [5, "e"]], ['ind', "state"])
  
  customers = spark.createDataFrame([["Alice", "2016-05-01", 50.00],
                                      ["Alice", "2016-05-03", 45.00],
                                      ["Alice", "2016-05-04", 55.00],
                                      ["Bob", "2016-05-01", 25.00],
                                      ["Bob", "2016-05-04", 29.00],
                                      ["Bob", "2016-05-06", 27.00]],
                                 ["name", "date", "amountSpent"])
  
  customers.show()
  
  window_spec = Window.partitionBy("name").orderBy("date").rowsBetween(-1, 1)
  
  result = customers.withColumn( "movingAvg", avg(customers["amountSpent"]).over(window_spec))
  
  result.show()
  </code></pre>
  
  <p>I am applying <code>avg</code> which is already built into <code>pyspark.sql.functions</code>, but if instead of <code>avg</code> I wanted to use something of more complicated and write my own function, how would I do that?</p>