title: Pyspark Dataframe: Get previous row that meets a condition
body: <p>For every row in a PySpark DataFrame I am trying to get a value from the first preceding row that satisfied a certain condition:</p>
  
  <p>That is if my dataframe looks like this:</p>
  
  <pre><code>X  | Flag
  1  | 1
  2  | 0
  3  | 0
  4  | 0
  5  | 1
  6  | 0
  7  | 0
  8  | 0
  9  | 1
  10 | 0
  </code></pre>
  
  <p>I want output that looks like this:</p>
  
  <pre><code>X  | Lag_X | Flag
  1  | NULL  | 1
  2  | 1     | 0
  3  | 1     | 0
  4  | 1     | 0
  5  | 1     | 1
  6  | 5     | 0
  7  | 5     | 0
  8  | 5     | 0
  9  | 5     | 1
  10 | 9     | 0
  </code></pre>
  
  <p>I thought I could do this with lag function and a WindowSpec, unfortunately WindowSpec doesnt support <code>.filter</code> or <code>.when</code>, so this does not work: </p>
  
  <pre><code>conditional_window = Window().orderBy(X).filter(df[Flag] == 1)
  df = df.withColumn('lag_x', f.lag(df[x],1).over(conditional_window)
  </code></pre>
  
  <p>It seems like this should be simple, but I have been racking my brain trying to find a solution so any help with this would be greatly appreciated</p>