title: Spark replace all NaNs to null in DataFrame API
body: <p>I have a dataframe with many double (and/or float) columns, which do contain NaNs. I want to replace all NaNs (i.e. Float.NaN and Double.NaN) with null. </p>
  
  <p>I can do this with e.g. for a single  column <code>x</code>:</p>
  
  <pre><code>val newDf = df.withColumn("x", when($"x".isNaN,lit(null)).otherwise($"x"))
  </code></pre>
  
  <p>This works but I'd like to do this for all columns at once. I recently discovered the <code>DataFrameNAFunctions</code> (<code>df.na</code>) <code>fill</code> which sounds exactely what I need. Unfortunately I failed to do the above.  <code>fill</code> should replace all NaNs and nulls with a given value, so I do:</p>
  
  <pre><code>df.na.fill(null.asInstanceOf[java.lang.Double]).show
  </code></pre>
  
  <p>which gives me a <code>NullpointerException</code></p>
  
  <p>There is also a promising <code>replace</code> method, but I cant even compile the code:</p>
  
  <pre><code>df.na.replace("x", Map(java.lang.Double.NaN -&gt; null.asInstanceOf[java.lang.Double])).show
  </code></pre>
  
  <p>strangely, this gives me </p>
  
  <pre><code>Error:(57, 34) type mismatch;
   found   : scala.collection.immutable.Map[scala.Double,java.lang.Double]
   required: Map[Any,Any]
  Note: Double &lt;: Any, but trait Map is invariant in type A.
  You may wish to investigate a wildcard type such as `_ &lt;: Any`. (SLS 3.2.10)
      df.na.replace("x", Map(java.lang.Double.NaN -&gt; null.asInstanceOf[java.lang.Double])).show
  </code></pre>