title: Why is groupBy() a lot faster than distinct() in pyspark?
body: <p>I have seen a lot of performance improvement in my pyspark code when I replaced <code>distinct()</code> on a spark data frame with <code>groupBy()</code>. But I failed to understand the reason behind it.
  The whole intention was to remove the row level duplicates from the dataframe.</p>
  
  <p>I tried Googling the implementation of <code>groupBy()</code> and <code>distinct()</code> in pyspark, but was unable to find it.</p>
  
  <p>Can somebody explain or point me in the right direction for the explanation?</p>