title: Pyspark merge multiple columns into a json column
body: <p>I asked the question a while back for python, but now I need to do the same thing in PySpark.</p>
  
  <p>I have a dataframe (df) like so:</p>
  
  <pre><code>|cust_id|address    |store_id|email        |sales_channel|category|
  -------------------------------------------------------------------
  |1234567|123 Main St|10SjtT  |idk@gmail.com|ecom         |direct  |
  |4567345|345 Main St|10SjtT  |101@gmail.com|instore      |direct  |
  |1569457|876 Main St|51FstT  |404@gmail.com|ecom         |direct  |
  </code></pre>
  
  <p>and I would like to combine the last 4 fields into one metadata field that is a json like so:</p>
  
  <pre><code>|cust_id|address    |metadata                                                                                     |
  -------------------------------------------------------------------------------------------------------------------
  |1234567|123 Main St|{'store_id':'10SjtT', 'email':'idk@gmail.com','sales_channel':'ecom', 'category':'direct'}   |
  |4567345|345 Main St|{'store_id':'10SjtT', 'email':'101@gmail.com','sales_channel':'instore', 'category':'direct'}|
  |1569457|876 Main St|{'store_id':'51FstT', 'email':'404@gmail.com','sales_channel':'ecom', 'category':'direct'}   |
  </code></pre>
  
  <p>Here's the code I used to do this in python:</p>
  
  <pre><code>cols = [
      'store_id',
      'store_category',
      'sales_channel',
      'email'
  ]
  
  df1 = df.copy()
  df1['metadata'] = df1[cols].to_dict(orient='records')
  df1 = df1.drop(columns=cols)
  </code></pre>
  
  <p>but I would like to translate this to PySpark code to work with a spark dataframe; I do NOT want to use pandas in Spark.</p>