title: Match string at end of line in file
body: <p>I have two files:</p>
  <ol>
  <li><p>$hashfile: Hashes and ./relative/path/to/file/names, both on one line separated by 2 spaces</p>
  </li>
  <li><p>$badfiles: ./relative/path/to/file/names that I need to find in $hashfile to get the corresponding hash</p>
  </li>
  </ol>
  <p>Here is an extract of a $hashfile:</p>
  <pre><code>c2c99b59f3303cafac85c2c6df6653cc  ./vm-mount.sh
  058a8fb0b9366f248be32b7390e94595  ./Jerusalem_Canon EOS R5_20210601_031.jpg~
  23eba1c54846de5244312047e2709f9a  ./rsync-back.sh
  ff3f08f7bf45f8e9ef8b33192db3ce9a  ./vm-backup.sh
  11e0d980f3b2219f65da97a0318e7dce  ./Jerusalem_Canon EOS R5_20210601_031.jpg
  49fb1fb660dce09acd87861a228c899d  ./vm-test.sh
  </code></pre>
  <p>Here is an example of $badfiles containing the search patterns:</p>
  <pre><code>./Jerusalem_Canon EOS R5_20210601_031.jpg
  ./file.txt
  </code></pre>
  <p>I need to search the $hashfile for the patterns inside $badfiles and write the matching lines containing the hash to a third file $new.</p>
  <p>So far I've used the following:</p>
  <pre><code>grep -Ff &quot;$badfiles&quot; &quot;$hashfile&quot; &gt; &quot;$new&quot;
  </code></pre>
  <p>However, this will match both:</p>
  <pre><code>058a8fb0b9366f248be32b7390e94595  ./Jerusalem_Canon EOS R5_20210601_031.jpg~
  11e0d980f3b2219f65da97a0318e7dce  ./Jerusalem_Canon EOS R5_20210601_031.jpg
  </code></pre>
  <p>I then added a $ at the end of each line in $badfiles and changed the grep command to:</p>
  <pre><code>grep -f &quot;$badfiles&quot; &quot;$hashfile&quot; &gt; &quot;$new&quot;
  </code></pre>
  <p>This worked on a small test folder, but I'm concerned that a pattern search that won't be interpreted as a fixed string can create havoc on large file systems. I have some 300,000+ file names and hashes, some of which use special characters like &quot;':,;&lt;&gt;()[]? - in short any character that a Linux ext4 and/or Windows NTFS file system will accept.</p>
  <p>Any ideas?</p>
  <p>EDIT: <strong>Solution</strong></p>
  <p>Apparently grep does not provide a simple solution for including a newline into a fixed string search. @anubhava offered the best solution using awk:</p>
  <pre><code>awk 'NR == FNR {a[$0]; next}
  {b=$0; sub(/^\S+\s+/, &quot;&quot;, b)}
  b in a' &quot;$badfiles&quot; &quot;$hashfile&quot; &gt; &quot;$new&quot;
  </code></pre>
  <p>Note: $badfiles, $hashfiles and $new are variables holding file names.</p>
  <p>The above syntax is best described <a href="https://backreference.org/2010/02/10/idiomatic-awk/" rel="nofollow noreferrer">here under &quot;Two-file processing&quot;</a>. <code>NR</code> holds the line numbers so far read from all files, whereas <code>FNR</code> holds the line numbers read so far from the current file. So when awk has finished reading $badfiles and reads the first line of $hashfile, <code>NR</code> holds the sum of all lines read so far, and <code>FNR</code> equals 1 since this is the first line of a new file. <code>{a[$0]; next}</code> reads the $badfiles file into an array, <code>; next</code> prevents the program from executing the subsequent conditions and actions until the entire $badfiles is read, that is, until <code>NR == FNR</code> is false.</p>
  <p>When reading $hashfile, <code>$0</code> (the line that's been read) is assigned to <code>b</code> (<code>b=$0</code>). <code>sub(/^\S+\s+/, &quot;&quot;, b)</code> substitutes one or more non-space characters (<code>\S+</code>) at the beginning of the line (<code>^</code>), followed by one or more space characters (<code>\s+</code>) by <code>&quot;&quot;</code> (empty string) in variable <code>b</code>. This then leaves only the ./path/to/file inside variable <code>b</code>.</p>
  <p>The last line <code>b in a' &quot;$badfiles&quot; &quot;$hashfile&quot; &gt; &quot;$new&quot;</code> looks if variable <code>b</code> is found in <code>a</code> and, if yes, copies the line in $hashfile to file $new. If all lines in $badfiles have a matching entry in $hashfile, the corresponding $hashfile lines with the hashes are copied to $new.</p>
  <p>Since the hash value before the file name is of a fixed length, the awk statement can be simplified as follows:</p>
  <pre><code>awk 'NR == FNR {a[$0]; next}
  {b=substr($0,35)}
  b in a' &quot;$badfiles&quot; &quot;$hashfile&quot; &gt; &quot;$new&quot;
  </code></pre>
  <p>The above <code>substr()</code> statement takes the input line <code>$0</code> and strips off the first 34 characters, counting from 1. The substring <code>b</code> then starts at position 35. This is much like substring extraction in bash, for example <code>${mystring:34}</code>. Note that bash substring extraction starts counting at 0.</p>
  <p>I use now a variation of that awk command to create a new hashfile containing all file hashes except for those listed in <code>$deletedfiles</code>:</p>
  <pre><code>awk 'NR == FNR {a[$0]; next}
  {b=substr($0,35)}
  !(b in a)' &quot;$deletedfiles&quot; &quot;$hashfile&quot; &gt; &quot;$new&quot;
  </code></pre>
  <p>With the above command, every string <code>b</code> (from $hashfile) NOT found in $deletedfiles copies the corresponding line from $hashfile to $new. One has to pay special attention to an empty $deletedfiles file: If $deletedfiles is an empty file, the $new file will be empty too! The expected result is that $new file is identical to the $hashfile.</p>
  <p>This solution works very well (and fast), even with 200,000-300,000 filenames in one hashfile.</p>