title: How to filter data using window functions in spark
body: <p>I have the following data :</p>
  
  <pre><code>rowid uid time code
     1  1      5    a
     2  1      6    b
     3  1      7    c
     4  2      8    a
     5  2      9    c
     6  2      9    c
     7  2     10    c
     8  2     11    a
     9  2     12    c
  </code></pre>
  
  <p>Now I wanted to filter the data in such a way that I can remove the rows 6 and 7 as for a particular uid i want to keep just one row with value 'c' in code</p>
  
  <p>So the expected data should be :</p>
  
  <pre><code>rowid uid time code
     1  1      5    a
     2  1      6    b
     3  1      7    c
     4  2      8    a
     5  2      9    c
     8  2     11    a
     9  2     12    c
  </code></pre>
  
  <p>I'm using window function something like this : </p>
  
  <pre><code>val window = Window.partitionBy("uid").orderBy("time")
  val change = ((lag("code", 1).over(window) &lt;=&gt; "c")).cast("int")
  </code></pre>
  
  <p>This would help us identify each row with a code 'c'. Can i extend this to filter out rows to get the expected data</p>