title: Replacing null values with 0 after spark dataframe left outer join
body: <p>I have two dataframes called <em>left</em> and <em>right</em>.</p>
  
  <pre><code>scala&gt; left.printSchema
  root
  |-- user_uid: double (nullable = true)
  |-- labelVal: double (nullable = true)
  |-- probability_score: double (nullable = true)
  
  scala&gt; right.printSchema
  root
  |-- user_uid: double (nullable = false)
  |-- real_labelVal: double (nullable = false)
  </code></pre>
  
  <p>Then, I join them to get the joined Dataframe. It is a <strong><em>left outer join</em></strong>. Anyone interested in the natjoin function can find it here.</p>
  
  <blockquote>
    <p><a href="https://gist.github.com/anonymous/f02bd79528ac75f57ae8">https://gist.github.com/anonymous/f02bd79528ac75f57ae8</a></p>
  </blockquote>
  
  <pre><code>scala&gt; val joinedData = natjoin(predictionDataFrame, labeledObservedDataFrame, "left_outer")
  
  scala&gt; joinedData.printSchema
  |-- user_uid: double (nullable = true)
  |-- labelVal: double (nullable = true)
  |-- probability_score: double (nullable = true)
  |-- real_labelVal: double (nullable = false)
  </code></pre>
  
  <p>Since it is a left outer join, the real_labelVal column has nulls when user_uid is not present in right.</p>
  
  <pre><code>scala&gt; val realLabelVal = joinedData.select("real_labelval").distinct.collect
  realLabelVal: Array[org.apache.spark.sql.Row] = Array([0.0], [null])
  </code></pre>
  
  <p><strong>I want to replace the null values in the realLabelVal column with 1.0.</strong></p>
  
  <p>Currently I do the following:</p>
  
  <ol>
  <li>I find the index of real_labelval column and use the spark.sql.Row API to set the nulls to 1.0.
  (This gives me a RDD[Row])</li>
  <li>Then I apply the schema of the joined dataframe to get the cleaned dataframe.</li>
  </ol>
  
  <p>The code is as follows: </p>
  
  <pre><code> val real_labelval_index = 3
   def replaceNull(row: Row) = {
      val rowArray = row.toSeq.toArray
       rowArray(real_labelval_index) = 1.0
       Row.fromSeq(rowArray)
   }
  
   val cleanRowRDD = joinedData.map(row =&gt; if (row.isNullAt(real_labelval_index)) replaceNull(row) else row)
   val cleanJoined = sqlContext.createDataFrame(cleanRowRdd, joinedData.schema)
  </code></pre>
  
  <p><strong>Is there an elegant or efficient way to do this?</strong></p>
  
  <p>Goolging hasn't helped much. 
  Thanks in advance.</p>