title: Python - remove duplicate entries from csv file
body: <p>I have a big <code>poems.csv</code> file with entries like this:</p>
  
  <pre><code>"
  this is a good poem. 
  ",1
  
  "  
  this is a bad poem.    
  ",0
  
  "
  this is a good poem. 
  ",1
  
  "  
  this is a bad poem.    
  ",0
  </code></pre>
  
  <p>and I would like to remove duplicates from it:</p>
  
  <p>If the file did not have the binary classifier I could just remove duplicate lines, like this:</p>
  
  <pre><code>with open(data_in,'r') as in_file, open(data_out,'w') as out_file:
      seen = set() # set for fast O(1) amortized lookup
      for line in in_file:
          if line in seen: continue # skip duplicate
          seen.add(line)
          out_file.write(line)
  </code></pre>
  
  <p>But this would remove all classification as well. How do I remove duplicate entries  keeping <code>0s</code> and <code>1s</code>?</p>
  
  <p>expected output:</p>
  
  <pre><code>"
  this is a good poem. 
  ",1
  
  "  
  this is a bad poem.    
  ",0
  </code></pre>