title: Filter array column content
body: <p>I am using pyspark 2.3.1 and would like to filter array elements with an expression and not an using udf:</p>
  
  <pre><code>&gt;&gt;&gt; df = spark.createDataFrame([(1, "A", [1,2,3,4]), (2, "B", [1,2,3,4,5])],["col1", "col2", "col3"])
  &gt;&gt;&gt; df.show()
  +----+----+---------------+
  |col1|col2|           col3|
  +----+----+---------------+
  |   1|   A|   [1, 2, 3, 4]|
  |   2|   B|[1, 2, 3, 4, 5]|
  +----+----+---------------+
  </code></pre>
  
  <p>The expreesion shown below is wrong, I wonder how to tell spark to remove out any values from the array in col3 which are smaller than 3. I want something like:</p>
  
  <pre><code>&gt;&gt;&gt; filtered = df.withColumn("newcol", expr("filter(col3, x -&gt; x &gt;= 3)")).show()
  &gt;&gt;&gt; filtered.show()
  +----+----+---------+
  |col1|col2|   newcol|
  +----+----+---------+
  |   1|   A|   [3, 4]|
  |   2|   B|[3, 4, 5]|
  +----+----+---------+
  </code></pre>
  
  <p>I have already an udf solution, but it is very slow (> 1 billions data rows):</p>
  
  <pre><code>largerThan = F.udf(lambda row,max: [x for x in row if x &gt;= max], ArrayType(IntegerType()))
  df = df.withColumn('newcol', size(largerThan(df.queries, lit(3))))
  </code></pre>
  
  <p>Any help is welcome. Thank you very much in advance.</p>