title: Fill Pyspark dataframe column null values with average value from same column
body: <p>With a dataframe like this, </p>
  
  <pre><code>rdd_2 = sc.parallelize([(0,10,223,"201601"), (0,10,83,"2016032"),(1,20,None,"201602"),(1,20,3003,"201601"), (1,20,None,"201603"), (2,40, 2321,"201601"), (2,30, 10,"201602"),(2,61, None,"201601")])
  
  df_data = sqlContext.createDataFrame(rdd_2, ["id", "type", "cost", "date"])
  df_data.show()
  
  +---+----+----+-------+
  | id|type|cost|   date|
  +---+----+----+-------+
  |  0|  10| 223| 201601|
  |  0|  10|  83|2016032|
  |  1|  20|null| 201602|
  |  1|  20|3003| 201601|
  |  1|  20|null| 201603|
  |  2|  40|2321| 201601|
  |  2|  30|  10| 201602|
  |  2|  61|null| 201601|
  +---+----+----+-------+
  </code></pre>
  
  <p>I need to fill the null values with the average of the existing values, with the expected result being</p>
  
  <pre><code>+---+----+----+-------+
  | id|type|cost|   date|
  +---+----+----+-------+
  |  0|  10| 223| 201601|
  |  0|  10|  83|2016032|
  |  1|  20|1128| 201602|
  |  1|  20|3003| 201601|
  |  1|  20|1128| 201603|
  |  2|  40|2321| 201601|
  |  2|  30|  10| 201602|
  |  2|  61|1128| 201601|
  +---+----+----+-------+
  </code></pre>
  
  <p>where <code>1128</code> is the average of the existing values. I need to do that for several columns. </p>
  
  <p>My current approach is to use <code>na.fill</code>:</p>
  
  <pre><code>fill_values = {column: df_data.agg({column:"mean"}).flatMap(list).collect()[0] for column in df_data.columns if column not in ['date','id']}
  df_data = df_data.na.fill(fill_values)
  
  +---+----+----+-------+
  | id|type|cost|   date|
  +---+----+----+-------+
  |  0|  10| 223| 201601|
  |  0|  10|  83|2016032|
  |  1|  20|1128| 201602|
  |  1|  20|3003| 201601|
  |  1|  20|1128| 201603|
  |  2|  40|2321| 201601|
  |  2|  30|  10| 201602|
  |  2|  61|1128| 201601|
  +---+----+----+-------+
  </code></pre>
  
  <p>But this is very cumbersome. Any ideas?</p>