title: why does filter remove null value by default on spark dataframe?
body: <p><code>filter</code> on basic scala collections containing <code>null</code> values has the following (and quite intuitive) behaviour:</p>
  
  <pre><code>scala&gt; List("a", "b", null).filter(_ != "a")
  res0: List[String] = List(b, null)
  </code></pre>
  
  <p>However, I was very surprised to figure out that the following filter removes nulls in spark dataframe:</p>
  
  <pre><code>scala&gt; val df = List(("a", null), ( "c", "d")).toDF("A", "B")
  scala&gt; df.show
  +---+----+
  |  A|   B|
  +---+----+
  |  a|null|
  |  c|   d|
  +---+----+
  scala&gt; df.filter('B =!= "d").show
  +---+---+
  |  A|  B|
  +---+---+
  +---+---+
  </code></pre>
  
  <p>If I want to keep <code>null</code> values, I should add</p>
  
  <pre><code>df.filter('B =!= "d" || 'B.isNull).show
  +---+----+
  |  A|   B|
  +---+----+
  |  a|null|
  +---+----+
  </code></pre>
  
  <p>Personally, I think that removing nulls by default is very error prone. 
  Why this choice? and why is not explicitely stated in the api documentation? Am I missing something? </p>