title: Bash: how to redirect continuous output to text file but make lines unique?
body: <p>Ok, I have this line that outputs data to a text file.  The only issue is I need the lines to be unique.  So, if it is going to add a line that already exists how can I prevent that?  This is my script:</p>
  
  <pre><code> tcpdump -lvi any "udp port 53" 2&gt;/dev/null|grep -E 'A\?'|awk '{print $(NF-1)}' &gt;&gt; /tmp/domains
  </code></pre>
  
  <p>Do I pipe it go awk and somehow delete duplicates?  Do I have another script run everyone minute that removes duplicates?</p>
  
  <p>Here is the output of loading up Amazon.com:</p>
  
  <pre><code> amazon.com.
   amazon.com.
   www.amazon.com.
   www.amazon.com.
   amazon.com.
   www.amazon.com.
   a0e90b2a1701074fb52d450dc80084cb1.labs.cloudfront.net.
   a0e90b2a1701074fb52d450dc80084cb1.labs.cloudfront.net.
   ad.doubleclick.net.
   ad.doubleclick.net.
   ecx.images-amazon.com.
   ...more
  </code></pre>
  
  <p>And in looking at my output it looks like I need to figure out why there is a trailing dot.</p>