title: In DataFrame.withColumn, how can I check if the column's value is null as a condition for the second parameter?
body: <p>If I have a DataFrame called df that looks like:</p>
  
  <pre><code>+----+----+
  |  a1+  a2|
  +----+----+
  | foo| bar|
  | N/A| baz|
  |null| etc|
  +----+----+
  </code></pre>
  
  <p>I can selectively replace values like so:</p>
  
  <pre><code>val df2 = df.withColumn("a1", when($"a1" === "N/A", $"a2"))
  </code></pre>
  
  <p>so that df2 looks like:</p>
  
  <pre><code>+----+----+
  |  a1+  a2|
  +----+----+
  | foo| bar|
  | baz| baz|
  |null| etc|
  +----+----+
  </code></pre>
  
  <p>but why can't I check if it's null, like:</p>
  
  <pre><code>val df3 = df2.withColumn("a1", when($"a1" === null, $"a2"))
  </code></pre>
  
  <p>so that I get:</p>
  
  <pre><code>+----+----+
  |  a1+  a2|
  +----+----+
  | foo| bar|
  | baz| baz|
  | etc| etc|
  +----+----+
  </code></pre>
  
  <p>Edit: $"a1".isNull doesn't seem to work.  Could it be because of how I'm constructing the dataframe I'm using to test, which is as follows?</p>
  
  <pre><code>val schema = StructType(
                  StructField("a1", StringType, false) ::
                  StructField("a2", StringType, false) :: Nil
  )
  
  val data = sc.parallelize(Array(
                  Row("foo","bar"),
                  Row("N/A","baz"),
                  Row(null,"etc"))
  )
  
  val df = sqlContext.createDataFrame(data, schema)
  </code></pre>
  
  <p>I also can't use coalesce, as far as I know, because sometimes I need to use a static value instead of another column's value.</p>
  
  <p>Edit again: Setting my test columns to nullable = false doesn't help.</p>