title: Standard unix tool for unique set
body: <p>The following python script prints the first occurrence of a line in a file.</p>
  
  <pre><code>#!/usr/bin/env python
  import sys
  
  x = set() 
  
  for line in sys.stdin:
      if line not in x:
          print line,
          x.add(line)
  </code></pre>
  
  <p><code>uniq</code> only works for adjacent lines. The file this will be used on is very large so <code>sort | uniq</code> is not ideal. Is there a standard tool that does this?</p>