title: PySpark UDF issues when referencing outside of function
body: <p>I facing the issue that I get the error</p>
  <blockquote>
  <p>TypeError: cannot pickle '_thread.RLock' object</p>
  </blockquote>
  <p>when I try to apply the following code:</p>
  <pre><code>from pyspark.sql.types import *
  from pyspark.sql.functions import *
  
  data_1 = [('James','Smith','M',30),('Anna','Rose','F',41),
    ('Robert','Williams','M',62), 
  ]
  data_2 = [('Junior','Smith','M',15),('Helga','Rose','F',33),
    ('Mike','Williams','M',77), 
  ]
  columns = [&quot;firstname&quot;,&quot;lastname&quot;,&quot;gender&quot;,&quot;age&quot;]
  df_1 = spark.createDataFrame(data=data_1, schema = columns)
  df_2 = spark.createDataFrame(data=data_2, schema = columns)
  
  def find_n_people_with_higher_age(x):
    return df_2.filter(df_2['age']&gt;=x).count()
  
  find_n_people_with_higher_age_udf = udf(find_n_people_with_higher_age, IntegerType())
  df_1.select(find_n_people_with_higher_age_udf(col('category_id')))
  </code></pre>