title: How to deal with null values in spark reduceByKey function?
body: <p>I have a spark DataFrame (df) which looks like this:</p>
  
  <pre><code>+----------+--------+----------+--------+                                                                 
  |        c1|      c2|        c3|      c4|
  +----------+--------+----------+--------+
  |      1   |    5   |      null|    7   |
  +----------+--------+----------+--------+
  |      1   |    5   |      4   |    8   |
  +----------+--------+----------+--------+
  |      1   |    3   |      null|   11   |
  +----------+--------+----------+--------+
  |      1   |    3   |      null| null   |
  +----------+--------+----------+--------+
  |      2   |    6   |      23  |   17   |
  +----------+--------+----------+--------+
  |      2   |    6   |      7   |    3   |
  +----------+--------+----------+--------+
  |      2   |    3   |      null|   11   |
  +----------+--------+----------+--------+
  |      2   |    3   |      null|   17   |
  +----------+--------+----------+--------+
  </code></pre>
  
  <p>I want to aggregate using <code>(c1,c2)</code> as key and have <code>average</code> of <code>c3</code> and <code>c4</code>, so that I have this:</p>
  
  <pre><code>+----------+--------+----------+--------+                                                                 
  |        c1|      c2|        c3|      c4|
  +----------+--------+----------+--------+
  |      1   |    5   |      4   |  7.5   |
  +----------+--------+----------+--------+
  |      1   |    3   |      null|   11   |
  +----------+--------+----------+--------+
  |      2   |    6   |      15  |    10  |
  +----------+--------+----------+--------+
  |      2   |    3   |      null|   14   |
  +----------+--------+----------+--------+
  </code></pre>
  
  <p>So, essentially I am ignoring the <code>null</code> values.</p>
  
  <p>My <em>half-baked</em> code looks something like this: </p>
  
  <pre><code>val df1 = df.
            // just working on c3 for time being
            map(x =&gt; ((x.getInt(0), x.getInt(1)), x.getDouble(3))).
            reduceByKey( 
              (x, y) =&gt; {
              var temp = 0
              var sum = 0.0
              var flag = false
              if (x == null) {
                if (y != null) {
                  temp = temp + 1
                  sum = y
                  flag = true
                }
              } else {
                if (y == null) {
                  temp = temp + 1
                  sum = x 
                } else {
                  temp = temp + 1
                  sum = x + y
                  flag = true
                } 
              } 
              if (flag == false) {
                null 
              } else {
                sum/temp 
              }
              }
            )
  </code></pre>
  
  <p>Obviously, the above code is not working. Any help to make the code work is very much appreciated.</p>
  
  <p><strong>Edit 1</strong> The answer given by @zero232 is a solution. However, it is not "the solution" I am looking for. My interest was to understand <em>how to deal with null values when writing a custom function for <code>reduceByKey()</code></em>. <strong>I am re-asking the question below</strong>:</p>
  
  <p>I want to aggregate using <code>(c1,c2)</code> as key and have <code>root mean square</code> [{sum(a_i^2)}^0.5] (or some function which is not available in spark for that matter) of <code>c3</code> and <code>c4</code> while ignoring the null values, so that I have this:</p>
  
  <pre><code>+----------+--------+----------+--------+                                                                 
  |        c1|      c2|        c3|      c4|
  +----------+--------+----------+--------+
  |      1   |    5   |      4   | 10.63  |
  +----------+--------+----------+--------+
  |      1   |    3   |      null|   11   |
  +----------+--------+----------+--------+
  |      2   |    6   |   24.04  |  17.26 |
  +----------+--------+----------+--------+
  |      2   |    3   |      null| 20.24  |
  +----------+--------+----------+--------+
  </code></pre>