title: pyspark dataframe to extract each distinct word from a column of string and put them into a new dataframe
body: <p>I am trying to find all strings in a column in pyspark dataframe.</p>
  <p>the input df:</p>
  <pre><code> id val 
   1  &quot;book bike car&quot;
   15 &quot;car TV bike&quot;
  </code></pre>
  <p>I need an output df like: (the word_index value is auto-increment index and the order of values in &quot;val_new&quot; is random.)</p>
  <pre><code> val_new word_index
   TV      1
   car     2
   bike    3
   book    4 
  </code></pre>
  <p>My code :</p>
  <pre><code> import pyspark.sql.functions as F
   from pyspark.sql.types import  ArrayType, StringType
   import re
  
   def my_f(col):
       if not col: 
           return 
       s = ''
       if isinstance(col, str):
           s = re.sub('[^a-zA-Z0-9]+', ' ', col).split()
       return s 
  
   my_udf = F.udf(my_f, ArrayType(StringType()))
  
   df = spark.createDataFrame([(1, 'book bike car'), (18, 'car TV bike')], ['id', 'val'])
   df = df.withColumn('val_new', my_udf(F.col('val')))
  </code></pre>
  <p>I have converted the string to array but how to extract the words from each row, remove duplicates, and create a new dataframe with the two new columns ?</p>
  <p>I do not want to use groupBy and aggregate because the dataframe may be large and I do not need the &quot;id&quot; column and any duplicates of &quot;val&quot;.</p>
  <p>thanks</p>