title: Most optimal way to removing Duplicates in pySpark
body: <p>I am trying to remove duplicates in spark dataframes by using dropDuplicates() on couple of columns. But job is getting hung due to lots of shuffling involved and data skew. I have used 5 cores and 30GB of memory to do this. Data on which I am performing dropDuplicates() is about 12 million rows.</p>
  
  <p>Please suggest me the most optimal way to remove duplicates in spark, considering data skew and shuffling involved.</p>