title: How to get distinct rows in dataframe using pyspark?
body: <p>I understand this is just a very simple question and most likely have been answered somewhere, but as a beginner I still don't get it and am looking for your enlightenment, thank you in advance:</p>
  
  <p>I have a interim dataframe:</p>
  
  <pre><code>+----------------------------+---+
  |host                        |day|
  +----------------------------+---+
  |in24.inetnebr.com           |1  |
  |uplherc.upl.com             |1  |
  |uplherc.upl.com             |1  |
  |uplherc.upl.com             |1  |
  |uplherc.upl.com             |1  |
  |ix-esc-ca2-07.ix.netcom.com |1  |
  |uplherc.upl.com             |1  |
  </code></pre>
  
  <p>What I need is to remove all the redundant items in host column, in another word, I need to get the final distinct result like:</p>
  
  <pre><code>+----------------------------+---+
  |host                        |day|
  +----------------------------+---+
  |in24.inetnebr.com           |1  |
  |uplherc.upl.com             |1  |
  |ix-esc-ca2-07.ix.netcom.com |1  |
  |uplherc.upl.com             |1  |
  </code></pre>