title: How to parse and explode a list of dictionaries stored as string in pyspark?
body: <p>I have some data that is stored in CSV. Sample data is available here - <a href="https://github.com/PranayMehta/apache-spark/blob/master/data.csv" rel="nofollow noreferrer">https://github.com/PranayMehta/apache-spark/blob/master/data.csv</a></p>
  <p>I read the data using pyspark</p>
  <pre><code>df = spark.read.csv(&quot;data.csv&quot;,header=True)
  df.printSchema()
  root
   |-- freeform_text: string (nullable = true)
   |-- entity_object: string (nullable = true)
  
  &gt;&gt;&gt; df.show(truncate=False)
  +---------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
  |freeform_text                    |entity_object                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
  +---------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
  |Grapes are good. Bananas are bad.|[{'name': 'Grapes', 'type': 'OTHER', 'salience': '0.8335162997245789', 'sentiment_score': '0.8999999761581421', 'sentiment_magnitude': '0.8999999761581421', 'metadata': {}, 'mentions': {'mention_text': 'Grapes', 'mention_type': 'COMMON'}}, {'name': 'Bananas', 'type': 'OTHER', 'salience': '0.16648370027542114', 'sentiment_score': '-0.8999999761581421', 'sentiment_magnitude': '0.8999999761581421', 'metadata': {}, 'mentions': {'mention_text': 'Bananas', 'mention_type': 'COMMON'}}]|
  |the weather is not good today    |[{'name': 'weather', 'type': 'OTHER', 'salience': '1.0', 'sentiment_score': '-0.800000011920929', 'sentiment_magnitude': '0.800000011920929', 'metadata': {}, 'mentions': {'mention_text': 'weather', 'mention_type': 'COMMON'}}]                                                                                                                                                                                                                                                                 |
  +---------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
  </code></pre>
  <p>Now, I want to explode and parse the fields in the <code>entity_object</code> column in this dataframe. Here is some more know-how on what this column contains -</p>
  <p>For every <code>freeform_text</code> stored in the Spark Dataframe, I have written some logic to parse out the entities using google's natural language API. These entities are stores as LIST of DICTIONARIES when I do the computation using pandas. I then convert them to string before storing them to Database.</p>
  <p>This CSV is what I read in spark dataframe as 2 columns - <code>freeform_text</code> and <code>entity_object</code>.</p>
  <p>The <code>entity_object</code> column as string is actually a LIST of dictionaries. It can be imagined as <code>LIST[ DICT1, DICT2 ]</code> and so on. So, some <code>entity_object</code> rows may have 1 element others may have more than 1 based on the number of entities in the output. For instance in the first row, there are 2 entities - <code>grapes</code> and <code>bananas</code>, whereas in 2nd row there is only entity <code>weather</code>.</p>
  <p>I want to explode this <code>entity_object</code> column so that 1 record of <code>freeform_text</code> can be exploded in multiple records.</p>
  <p>Here is a screenshot of how I would like my output to be -</p>
  <p><a href="https://i.stack.imgur.com/ZF6jT.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/ZF6jT.png" alt="enter image description here" /></a></p>