title: Pyspark Dataframe get unique elements from column with string as list of elements
body: <p>I have a dataframe (which is created by loading from multiple blobs in azure) where I have a column which is list of IDs. 
  Now, I want a list of unique IDs from this entire column:</p>
  
  <p>Here is an example - </p>
  
  <pre><code>df - 
  | col1 | col2 | col3  |
  | "a"  | "b"  |"[q,r]"|
  | "c"  | "f"  |"[s,r]"|
  </code></pre>
  
  <p>Here is my expected response: </p>
  
  <pre><code>resp = [q, r, s]
  </code></pre>
  
  <p>Any idea how to get there? </p>
  
  <p>My current approach is to convert the strings in col3 to python lists and then maybe flaten them out somehow. </p>
  
  <p>But so far I am not able to do so. I tried using user defined functions in pyspark but they only return strings and not lists. </p>
  
  <p>FlatMaps only work on RDD not on Dataframes so they are out of picture. </p>
  
  <p>Maybe there is way where I can specify this during the conversion from RDD to dataframe. But not sure how to do that. </p>