title: Find 3 closest dates in Spark dataframe per some criteria using pyspark
body: <p>I have the following spark dataframes:-</p>
  
  <p>df1</p>
  
  <pre><code>id  dia_date
  1   2/12/17
  1   4/25/16
  2   12/8/17
  2   6/12/11
  </code></pre>
  
  <p>df2</p>
  
  <pre><code>id  obs_date    obs_value
  1   2/16/17 4
  1   2/20/17 2
  1   2/9/17  4
  1   12/12/18    5
  1   4/18/16 1
  1   4/18/16 6
  1   4/30/16 7
  1   5/25/16 9
  2   12/12/17    10
  2   12/6/17 11
  2   12/14/17    4
  2   6/11/11 5
  2   6/11/11 6
  </code></pre>
  
  <p>I want the resulting dataframe to be as follows:-</p>
  
  <p>1) Find the three nearest date in by comparing the date in df1 in df2.<br>
  2) If it does not have 3 nearest dates insert null for them.<br>
  3) The nearest dates should be found grouped by the id field only. Like for dia_date for id '1' we must look at the obs_date field in the df2 for id1 '1' only.</p>
  
  <p>Example of resulting dataframe:-</p>
  
  <pre><code>id  dia_date    obs_date1   obs_val1    obs_date2   obs_val2    obs_date3   obs_val3
  1   2/12/17 2/9/17  4   2/16/17 4   2/20/17 2
  1   4/25/16 4/18/16 1   4/18/16 6   4/30/16 7
  2   12/8/17 12/6/17 11  12/12/17    10  12/14/17    4
  2   6/12/11 6/11/11 5   6/11/11 6   null    null
  </code></pre>
  
  <p>I want to do it using pyspark. Have tried some ways but finding it really difficult since I am just starting with pyspark.</p>