title: How to split a column that contains multiple key-value pairs into different columns in pyspark
body: <p>I am working on a very large dataset called Reddit on AWS. I have read a small sample first by :</p>
  
  <pre><code>file_lzo = sc.newAPIHadoopFile("s3://mv559/reddit/sample-data/", 
                                 "com.hadoop.mapreduce.LzoTextInputFormat", 
                                 "org.apache.hadoop.io.LongWritable", 
                                 "org.apache.hadoop.io.Text")
  </code></pre>
  
  <p>So I got a rdd called <code>file_lzo</code>. I toke the first element and the data looks like:</p>
  
  <pre><code>[(0,
    '{"archived":false,"author":"TistedLogic","author_created_utc":1312615878,"author_flair_background_color":null,"author_flair_css_class":null,"author_flair_richtext":[],"author_flair_template_id":null,"author_flair_text":null,"author_flair_text_color":null,"author_flair_type":"text","author_fullname":"t2_5mk6v","author_patreon_flair":false,"body":"Is it still r\\/BoneAppleTea worthy if it\'s the opposite?","can_gild":true,"can_mod_post":false,"collapsed":false,"collapsed_reason":null,"controversiality":0,"created_utc":1538352000,"distinguished":null,"edited":false,"gilded":0,"gildings":{"gid_1":0,"gid_2":0,"gid_3":0},"id":"e6xucdd","is_submitter":false,"link_id":"t3_9ka1hp","no_follow":true,"parent_id":"t1_e6xu13x","permalink":"\\/r\\/Unexpected\\/comments\\/9ka1hp\\/jesus_fking_woah\\/e6xucdd\\/","removal_reason":null,"retrieved_on":1539714091,"score":2,"send_replies":true,"stickied":false,"subreddit":"Unexpected","subreddit_id":"t5_2w67q","subreddit_name_prefixed":"r\\/Unexpected","subreddit_type":"public"}')]
  </code></pre>
  
  <p>Then I create a dataframe from this rdd by using</p>
  
  <pre><code>df = spark.createDataFrame(file_lzo,['idx','map_col'])
  df.show(4)
  </code></pre>
  
  <p>It looks like this</p>
  
  <pre><code>+-----+--------------------+
  |  idx|             map_col|
  +-----+--------------------+
  |    0|{"archived":false...|
  |70139|{"archived":false...|
  |70139|{"archived":false...|
  |70139|{"archived":false...|
  +-----+--------------------+
  only showing top 4 rows
  </code></pre>
  
  <p>And finally I want to get the data in dataframe format which looks like this, and save it as a parquet format in S3 for the future steps.</p>
  
  <p><img src="https://i.stack.imgur.com/w1Nnz.png" alt="the desired results"></p>
  
  <p>I have tried to create a Schema and then use <code>read.json</code>, however I got all of the value is Null</p>
  
  <pre><code>fields = [StructField("archived", BooleanType(), True), 
            StructField("author", StringType(), True),
            StructField("author_flair_css_class", StringType(), True),
            StructField("author_flair_text", StringType(), True),
            StructField("body", StringType(), True),
            StructField("can_gild", BooleanType(), True),         
            StructField("controversiality", LongType(), True),
            StructField("created_utc", StringType(), True),
            StructField("distinguished", StringType(), True),
            StructField("edited", StringType(), True),
            StructField("gilded", LongType(), True), 
            StructField("id", StringType(), True),
            StructField("is_submitter", StringType(), True),
            StructField("link_id", StringType(), True),
            StructField("parent_id", StringType(), True),
            StructField("permalink", StringType(), True),
            StructField("permalink", StringType(), True),
            StructField("removal_reason", StringType(), True),
            StructField("retrieved_on", LongType(), True), 
            StructField("score",LongType() , True),
            StructField("stickied", BooleanType(), True),  
            StructField("subreddit", StringType(), True),
            StructField("subreddit_id", StringType(), True)]
  
  schema = StructType(fields)
  
  +--------+------+----------------------+-----------------+----+--------+----------------+-----------+-------------+------+------+----+------------+-------+---------+---------+---------+--------------+------------+-----+--------+---------+------------+
  |archived|author|author_flair_css_class|author_flair_text|body|can_gild|controversiality|created_utc|distinguished|edited|gilded|  id|is_submitter|link_id|parent_id|permalink|permalink|removal_reason|retrieved_on|score|stickied|subreddit|subreddit_id|
  +--------+------+----------------------+-----------------+----+--------+----------------+-----------+-------------+------+------+----+------------+-------+---------+---------+---------+--------------+------------+-----+--------+---------+------------+
  |    null|  null|                  null|             null|null|    null|            null|       null|         null|  null|  null|null|        null|   null|     null|     null|     null|          null|        null| null|    null|     null|        null|
  |    null|  null|                  null|             null|null|    null|            null|       null|         null|  null|  null|null|        null|   null|     null|     null|     null|          null|        null| null|    null|     null|        null|
  |    null|  null|                  null|             null|null|    null|            null|       null|         null|  null|  null|null|        null|   null|     null|     null|     null|          null|        null| null|    null|     null|        null|
  +--------+------+----------------------+-----------------+----+--------+----------------+-----------+-------------+------+------+----+------------+-------+---------+---------+---------+--------------+------------+-----+--------+---------+------------+
  </code></pre>