title: PySpark and Protobuf Deserialization UDF Problem
body: <p>I'm getting this error</p>
  
  <blockquote>
    <p><code>Can't pickle &lt;class 'google.protobuf.pyext._message.CMessage'&gt;: it's not found as google.protobuf.pyext._message.CMessage</code> </p>
  </blockquote>
  
  <p>when I try to create a UDF in PySpark. Apparently, it uses CloudPickle to serialize the command however, I'm aware that protobuf messages contains <code>C++</code> implementations, meaning it cannot be pickled.</p>
  
  <p>I've tried finding a way to perhaps override <code>CloudPickleSerializer</code>, however, I wasn't able to find a way.</p>
  
  <p>Here's my example code:</p>
  
  <pre class="lang-py prettyprint-override"><code>from MyProject.Proto import MyProtoMessage
  from google.protobuf.json_format import MessageToJson
  import pyspark.sql.functions as F
  
  def proto_deserialize(body):
    msg = MyProtoMessage()
    msg.ParseFromString(body)
    return MessageToJson(msg)
  
  from_proto = F.udf(lambda s: proto_deserialize(s))
  
  base.withColumn("content", from_proto(F.col("body")))
  </code></pre>
  
  <p>Thanks in advance.</p>