title: How to use Pandas UDF Functionality in pyspark
body: <p>I have a spark frame with two columns which looks like:</p>
  
  <pre><code>+-------------------------------------------------------------+------------------------------------+
  |docId                                                        |id                                  |
  +-------------------------------------------------------------+------------------------------------+
  |DYSDG6-RTB-91d663dd-949e-45da-94dd-e604b6050cb5-1537142434000|91d663dd-949e-45da-94dd-e604b6050cb5|
  |VAVLS7-RTB-8e2c1917-0d6b-419b-a59e-cd4acc255bb7-1537142445000|8e2c1917-0d6b-419b-a59e-cd4acc255bb7|
  |VAVLS7-RTB-c818dcde-7a68-4c1e-9cc4-c841660732d2-1537146854000|c818dcde-7a68-4c1e-9cc4-c841660732d2|
  |IW2BYL-RTB-E9727F7D-D1BA-479C-9D3A-931F87E78B0A-1537146572000|E9727F7D-D1BA-479C-9D3A-931F87E78B0A|
  |DYSDG6-RTB-f50f79e9-3ec3-4bd8-8e53-f62c3f80bcb0-1537146220000|f50f79e9-3ec3-4bd8-8e53-f62c3f80bcb0|
  +-------------------------------------------------------------+------------------------------------+
  </code></pre>
  
  <p>I have a function that convert the id column into an 85 bit encoded string :</p>
  
  <pre><code>def convert_id(id):
      import base64 as bs
      id_str = str(id).replace("-", "") 
      return str(bs.a85encode(bytearray.fromhex(id_str)))[2:-1]
  </code></pre>
  
  <p>I want to transform this using pandas udf which is reported to be faster than the normal udf's.</p>
  
  <p>How can I achieve this ?
  TIA.</p>