title: Remove an element from a Python list of lists in PySpark DataFrame
body: <p>I am trying to remove an element from a Python list of lists:</p>
  
  <pre class="lang-none prettyprint-override"><code>+---------------+
  |        sources|
  +---------------+
  |           [62]|
  |        [7, 32]|
  |           [62]|
  |   [18, 36, 62]|
  |[7, 31, 36, 62]|
  |    [7, 32, 62]|
  </code></pre>
  
  <p>I want to be able to remove an element, <code>rm</code>, from each of the lists in the list above. I wrote a function that can do that for a list of lists:</p>
  
  <pre><code>def asdf(df, rm):
      temp = df
      for n in range(len(df)):
          temp[n] = [x for x in df[n] if x != rm]
      return(temp)
  </code></pre>
  
  <p>which does remove <code>rm = 1</code>:</p>
  
  <pre><code>a = [[1,2,3],[1,2,3,4],[1,2,3,4,5]]
  In:  asdf(a,1)
  Out: [[2, 3], [2, 3, 4], [2, 3, 4, 5]]
  </code></pre>
  
  <p>But I can't get it to work for a DataFrame:</p>
  
  <pre><code>asdfUDF = udf(asdf, ArrayType(IntegerType()))
  
  In: df.withColumn("src_ex", asdfUDF("sources", 32))
  
  Out: Py4JError: An error occurred while calling z:org.apache.spark.sql.functions.col. Trace:
  py4j.Py4JException: Method col([class java.lang.Integer]) does not exist
  </code></pre>
  
  <p>Desired behavior:</p>
  
  <pre><code>In: df.withColumn("src_ex", asdfUDF("sources", 32))
  Out: 
  
  +---------------+
  |         src_ex|
  +---------------+
  |           [62]|
  |            [7]|
  |           [62]|
  |   [18, 36, 62]|
  |[7, 31, 36, 62]|
  |        [7, 62]|
  </code></pre>
  
  <p>(except have the new column above appended to a PySpark DataFrame, <code>df</code>)</p>
  
  <p>Any suggestions or ideas?</p>