title: pyspark: get unique items in each column of a dataframe
body: <p>I have a spark dataframe containing 1 million rows and 560 columns. I need to find the count of unique items in each column of the dataframe.
  I have written the following code to achieve this but it is getting stuck and taking too much time to execute:</p>
  
  <pre><code>count_unique_items=[]
  
  for j in range(len(cat_col)):
      var=cat_col[j]
      count_unique_items.append(data.select(var).distinct().rdd.map(lambda r:r[0]).count())
  </code></pre>
  
  <h2>cat_col contains the column names of all the categorical variables</h2>
  
  <p>Is there any way to optimize this?</p>