title: How to merge list of list into single list in pyspark
body: <p>In spark dataframe, I have 1 column that contain list of list as rows. I want to merge the list of strings into one.</p>
  
  <pre><code>INPUT DATAFRAME:
  +-------+--------------------+
  | name  |friends             |
  +-------+--------------------+
  | Jim   |[["C","A"]["B","C"]]|
  +-------+--------------------+
  | Bill  |[["E","A"]["F","L"]]|
  +-------+--------------------+
  | Kim   |[["C","K"]["L","G"]]| 
  +-------+--------------------+
  
  OUTPUT DATAFRAME:  
  
  +-------+--------------------+
  | name  |friends             |
  +-------+--------------------+
  | Jim   |["C","A","B"]       |
  +-------+--------------------+
  | Bill  |["E","A","F","L"]   |
  +-------+--------------------+
  | Kim   |["C","K","L","G"]   | 
  +-------+--------------------+
  </code></pre>
  
  <p>I want to merge the list of list into single list and remove the duplicates as well.
  Thanks in advance</p>