text: user6271: I have two very large vcf files 2GB and 6GB I want to look for unique combinations of CHROM and POS and output the row that matches. However, because the files a so large my machine always hang and stop processing. Is there are a way to work around the problem of this large file? 
  I am using this command, taken from an answer to another question:
  awk  '{ if(NR==FNR){a[$1$2]=$0}else{if($1$2 in a){print}}}' file1.vcf file2.vcf
  
  file 1
  #CHROM  POS ID  REF ALT QUAL
  1   10366   rs58108140  G   A   
  1   10611   rs189107123 C   G   
  1   51954   rs185832753 G   C   
  1   13327   rs144762171 G   C
  
  file 2
  #CHROM  POS ID  REF ALT QUAL    
  1   10366   rs58108140  G   A   
  1   51935   rs181754315 C   T   
  1   51954   rs185832753 G   C   
  1   52058   rs62637813  G   C   
  1   52144   rs190291950 T   A
  
  output
  1 10366 rs58108140  G   A
  1 51954  rs185832753 G   C 
  
  
  user1461: Maybe comm, which is part of GNU coreutils, is more efficient.
  
  comm writes to standard output lines that are common, and lines that are unique, to two input files; a file name of ‘-’ means standard input. Synopsis:
  comm [option]… file1 file2
  
  In your case you could sort both files based on the first 3 columns:
  file1=file1.vcf
  file2=file2.vcf
  
  comm -12 <(awk '{print $1,$2,$3}' $file1 | sort) \
           <(awk '{print $1,$2,$3}' $file2 | sort) > matches
  
  
  # result:
  cat matches
  1 10366 rs58108140
  1 51954 rs185832753
  #CHROM POS ID
  
  This could be of course polished by removing the header.
  Also if you want the original lines as output, you can grep the matches to the original files:
  grep -f matches file1
  #CHROM POS ID REF ALT QUAL 
  1 10366 rs58108140 G A
  1 51954 rs185832753 G C
  
  Edit: In the first version, intermediate temp files were used, but as suggested by terdon in the comments, this can be avoided using Process Substitution allowing the <(..) contstruct.
  
original_text: 6271: <p>I have two very large vcf files 2GB and 6GB I want to look for unique combinations of CHROM and POS and output the row that matches. However, because the files a so large my machine always hang and stop processing. Is there are a way to work around the problem of this large file? </p>
  
  <p>I am using this command, taken from an answer to <a href="https://unix.stackexchange.com/q/543633/22222">another question</a>:</p>
  
  <pre><code>awk  '{ if(NR==FNR){a[<span class="math-container">$1$</span>2]=<span class="math-container">$0}else{if($1$</span>2 in a){print}}}' file1.vcf file2.vcf
  </code></pre>
  
  <p>file 1</p>
  
  <pre><code>#CHROM  POS ID  REF ALT QUAL
  1   10366   rs58108140  G   A   
  1   10611   rs189107123 C   G   
  1   51954   rs185832753 G   C   
  1   13327   rs144762171 G   C
  </code></pre>
  
  <p>file 2</p>
  
  <pre><code>#CHROM  POS ID  REF ALT QUAL    
  1   10366   rs58108140  G   A   
  1   51935   rs181754315 C   T   
  1   51954   rs185832753 G   C   
  1   52058   rs62637813  G   C   
  1   52144   rs190291950 T   A
  </code></pre>
  
  <p>output</p>
  
  <pre><code>1 10366 rs58108140  G   A
  1 51954  rs185832753 G   C 
  </code></pre>
   1461: <p>Maybe <a href="https://www.gnu.org/software/coreutils/manual/html_node/comm-invocation.html" rel="nofollow noreferrer">comm</a>, which is part of GNU coreutils, is more efficient.</p>
  <blockquote>
  <p>comm writes to standard output lines that are common, and lines that are unique, to two input files; a file name of ‘-’ means standard input. Synopsis:</p>
  <p>comm [option]… file1 file2</p>
  </blockquote>
  <p>In your case you could sort both files based on the first 3 columns:</p>
  <pre><code>file1=file1.vcf
  file2=file2.vcf
  
  comm -12 &lt;(awk '{print <span class="math-container">$1,$</span>2,<span class="math-container">$3}' $</span>file1 | sort) \
           &lt;(awk '{print <span class="math-container">$1,$</span>2,<span class="math-container">$3}' $</span>file2 | sort) &gt; matches
  
  
  # result:
  cat matches
  1 10366 rs58108140
  1 51954 rs185832753
  #CHROM POS ID
  </code></pre>
  <p>This could be of course polished by removing the header.
  Also if you want the original lines as output, you can grep the matches to the original files:</p>
  <pre><code>grep -f matches file1
  #CHROM POS ID REF ALT QUAL 
  1 10366 rs58108140 G A
  1 51954 rs185832753 G C
  </code></pre>
  <p>Edit: In the first version, intermediate temp files were used, but as suggested by terdon in the comments, this can be avoided using <a href="http://tldp.org/LDP/abs/html/process-sub.html" rel="nofollow noreferrer">Process Substitution</a> allowing the <code>&lt;(..)</code> contstruct.</p>
  