title: Fastest way -- Appending a line to a file only if it does not already exist
body: <p>given this question <a href="https://stackoverflow.com/questions/3557037/appending-a-line-to-a-file-only-if-it-does-not-already-exist">Appending a line to a file only if it does not already exist</a></p>
  
  <p>is there a faster way than the solution provided by @drAlberT?</p>
  
  <pre><code>grep -q -F 'string' foo.bar || echo 'string' &gt;&gt; foo.bar
  </code></pre>
  
  <p>I have implemented the above solution and I have to iterate it over a 500k lines file (i.e. check if a line is not already in a 500k lines set). Moreover, I've to run this process for a lot of times, maybe 10-50 million times. Needless to say it's kind of slow as it takes 25-30ms to run on my server (so 3-10+ days of runtime in total).</p>
  
  <p><strong>EDIT:</strong> the flow is the following: I have a file with 500k lines, each time I run, I get maybe 10-30 new lines and I check if they are already there or not. If not I add them, then I repeat many times. The order of my 500k lines files is important as I'm going through it with another process.</p>
  
  <p><strong>EDIT2</strong>: the 500k lines file is always containing unique lines, and I only care about "full lines", no substrings.</p>
  
  <p>Thanks a lot!</p>