title: Extracting URL parameters using Python and PySpark
body: <p>Say I have a column filled with URLs like in the following:</p>
  
  <pre><code>+------------------------------------------+
  |url                                       |
  +------------------------------------------+
  |https://www.example1.com?param1=1&amp;param2=a|
  |https://www.example2.com?param1=2&amp;param2=b|
  |https://www.example3.com?param1=3&amp;param2=c|
  +------------------------------------------+
  </code></pre>
  
  <p>What would be the best way of extracting the URL parameters from this column and adding them as columns to the dataframe to produce the below?</p>
  
  <pre><code>+-------------------------------------------+---------------+
  |                                        url| param1| param2|
  +-------------------------------------------+---------------+
  |https://www.example1.com?param1=1&amp;param2=a |      1|      a|
  |https://www.example2.com?param1=2&amp;param2=b |      2|      b|
  |https://www.example3.com?param1=3&amp;param2=c |      3|      c|
  |etc...                                     | etc...| etc...|
  +-------------------------------------------+---------------+
  </code></pre>
  
  <h3>My Attempts</h3>
  
  <p>I can think of two possible methods of doing this, using <code>functions.regexp_extract</code> from the pyspark library or by using <code>urllib.parse.parse_qs</code> and <code>urllib.parse.urlparse</code> from the standard library. The former solution uses regex which is a finicky method of extracting parameters from strings but the latter would need to be wrapped in a UDF to be used.</p>
  
  <pre class="lang-python prettyprint-override"><code>from pyspark.sql import *
  from pyspark.sql import functions as fn
  
  df = spark.createDataFrame(
    [
      ("https://www.example.com?param1=1&amp;param2=a",),
      ("https://www.example2.com?param1=2&amp;param2=b",),
      ("https://www.example3.com?param1=3&amp;param2=c",)
    ],
    ["url"]
  )
  </code></pre>
  
  <p>Regex solution:</p>
  
  <pre class="lang-python prettyprint-override"><code>df2 = df.withColumn("param1", fn.regexp_extract('url', 'param1=(\d)', 1))
  df2 = df2.withColumn("param2", fn.regexp_extract('url', 'param2=([a-z])', 1))
  df2.show()
  
  &gt;&gt; +------------------------------------------+------+------+
  &gt;&gt; |url                                       |param1|param2|
  &gt;&gt; +------------------------------------------+------+------+
  &gt;&gt; |https://www.example1.com?param1=1&amp;param2=a|1     |a     |
  &gt;&gt; |https://www.example2.com?param1=2&amp;param2=b|2     |b     |
  &gt;&gt; |https://www.example3.com?param1=3&amp;param2=c|3     |c     |
  &gt;&gt; +------------------------------------------+------+------+
  </code></pre>
  
  <p>UDF solution:</p>
  
  <pre class="lang-python prettyprint-override"><code>from urllib.parse import urlparse, parse_qs
  from pyspark.sql.types import MapType, StringType
  extract_params = udf(lambda x: {k: v[0] for k, v in parse_qs(urlparse(x).query).items()}, MapType(StringType(), StringType()))
  
  df3 = df.withColumn(
    "params", extract_params(df.url)
  )
  
  df3.withColumn(
    "param1", df3.params['param1']
  ).withColumn(
    "param2", df3.params['param2']
  ).drop("params").show()
  
  &gt;&gt; +------------------------------------------+------+------+
  &gt;&gt; |url                                       |param1|param2|
  &gt;&gt; +------------------------------------------+------+------+
  &gt;&gt; |https://www.example1.com?param1=1&amp;param2=a|1     |a     |
  &gt;&gt; |https://www.example2.com?param1=2&amp;param2=b|2     |b     |
  &gt;&gt; |https://www.example3.com?param1=3&amp;param2=c|3     |c     |
  &gt;&gt; +------------------------------------------+------+------+
  </code></pre>
  
  <p>I'd like to use the versatility of a library like <code>urllib</code> but would also like the optimisability of writing it in pyspark functions. Is there a better method than the two I've tried so far?</p>