title: python spark alternative to explode for very large data
body: <p>I have a dataframe like this:</p>
  
  <pre><code>df = spark.createDataFrame([(0, ["B","C","D","E"]),(1,["E","A","C"]),(2, ["F","A","E","B"]),(3,["E","G","A"]),(4,["A","C","E","B","D"])], ["id","items"])
  </code></pre>
  
  <p>which creates a data frame <code>df</code> like this:</p>
  
  <pre><code>+---+-----------------+
  |  0|     [B, C, D, E]|
  |  1|        [E, A, C]|
  |  2|     [F, A, E, B]|
  |  3|        [E, G, A]|
  |  4|  [A, C, E, B, D]|
  +---+-----------------+ 
  </code></pre>
  
  <p>I would like to get a result like this:</p>
  
  <pre><code>+---+-----+
  |all|count|
  +---+-----+
  |  F|    1|
  |  E|    5|
  |  B|    3|
  |  D|    2|
  |  C|    3|
  |  A|    4|
  |  G|    1|
  +---+-----+
  </code></pre>
  
  <p>Which essentially just finds all distinct elements in <code>df["items"]</code> and counts their frequency. If my data was of a more manageable size, I would just do this:</p>
  
  <pre><code>all_items = df.select(explode("items").alias("all")) 
  result = all_items.groupby(all_items.all).count().distinct() 
  result.show()
  </code></pre>
  
  <p>But because my data has millions of rows and thousands of elements in each list, this is not an option. I was thinking of doing this row by row, so that I only work with 2 lists at a time. Because most elements are frequently repeated in many rows (but the list in each row is a set), this approach should solve my problem. But the problem is, I don't really know how to do this in Spark, as I've only just started learning it. Could anyone help, please?</p>