title: How to deal with Spark UDF input/output of primitive nullable type
body: <p>The issues:</p>
  
  <p>1) Spark doesn't call UDF if input is column of primitive type that contains <code>null</code>:</p>
  
  <pre><code>inputDF.show()
  
  +-----+
  |  x  |
  +-----+
  | null|
  |  1.0|
  +-----+
  
  inputDF
    .withColumn("y",
       udf { (x: Double) =&gt; 2.0 }.apply($"x") // will not be invoked if $"x" == null
    )
    .show()
  
  +-----+-----+
  |  x  |  y  |
  +-----+-----+
  | null| null|
  |  1.0|  2.0|
  +-----+-----+
  </code></pre>
  
  <p>2) Can't produce <code>null</code> from UDF as a column of primitive type:</p>
  
  <p><code>udf { (x: String) =&gt; null: Double } // compile error</code></p>