title: Pyspark: PicklingError: Could not serialize object:
body: <p>I have the following two data frames: df_whitelist and df_text</p>
  
  <pre><code>+-------+--------------------+
  |keyword|    whitelist_terms |
  +-------+--------------------+
  |     LA|             LA city|
  |     LA|        US LA in da |
  | client|this client has i...|
  | client|our client has do...|
  +-------+--------------------+
  +--------------------+----------+
  |                Text|  Keywords|
  +--------------------+----------+
  |the client as ada...|client;ada|
  |this client has l...| client;LA|
  +--------------------+----------+
  </code></pre>
  
  <p>In df_whitelist, each keyword corresponds to a set of terms, e.g. keyword LA corresponds to “LA city” and “US LA in da”. 
  In df_text, I have text and some keywords found in this text.
  What I want to do is that for each piece of text, such as “the client has ada..”, for each of its keywords “client” and “ada”, check through all the whitelist terms for that keyword, to see how many times the term occurred in the text.
  what I have tried is like following:</p>
  
  <pre><code>import pyspark.sql.functions as F
  import pyspark.sql.types as T
  import re
  def whitelisting(text,listOfKeyword,df_whitelist):
      keywords = listOfKeyword.split(";")
      found_whiteterms_count = 0
      for k in keywords:
          if df_whitelist.filter(df_whitelist.keyword == k).count() == 0:
              found_whiteterms_count = found_whiteterms_count + 0
          else:
              df = df_whitelist.filter(df_whitelist.keyword == k).select("whitelist_terms")
              n = df.rdd.map(lambda x:len(re.findall(x["whitelist_terms"],text))).reduce(lambda x, y: x+y)
              found_whiteterms_count = found_whiteterms_count + n    
      return found_whiteterms_count     
  whitelisting_udf = F.udf(lambda text,listOfKeyword: whitelisting(text,listOfKeyword,df_whitelist),T.IntegerType())
  text.withColumn("whitelist_counts", whitelisting_udf(text.Text,text.Keywords))
  </code></pre>
  
  <p>and I got the error:</p>
  
  <pre><code>PicklingError: Could not serialize object: Py4JError: An error occurred while calling o1153.__getstate__. Trace:
  py4j.Py4JException: Method __getstate__([]) does not exist
      at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:318)
      at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:326)
      at py4j.Gateway.invoke(Gateway.java:272)
      at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
      at py4j.commands.CallCommand.execute(CallCommand.java:79)
      at py4j.GatewayConnection.run(GatewayConnection.java:214)
      at java.base/java.lang.Thread.run(Thread.java:844)
  </code></pre>
  
  <p>I can not figure out after trying for some time. Could anybody kindly help to point out the problem and how to fix it. Thanks.</p>