title: How to remove nulls with array_remove Spark SQL Built-in Function
body: <p>Spark 2.4 introduced new useful Spark SQL functions involving arrays but I was a little bit puzzled when I find out that the result of:
  <code>select array_remove(array(1, 2, 3, null, 3), null)</code> is <code>null</code> and not <code>[1, 2, 3, 3].</code></p>
  
  <p>Is this an expected behavior? Is it possible to remove nulls using <code>array_remove</code>?</p>
  
  <p>As a side note, for now the alternative I am using is a higher order function in databricks:</p>
  
  <p><code>select filter(array(1, 2, 3, null, 3), x -&gt; x is not null)</code></p>