text: user982: I have a reference database with contains 100s of sequences in fasta format. Some of these sequences have duplicate names like so:
  >1_uniqueGeneName
  atgc
  >1_anotherUniqueGeneName
  atgc
  >1_duplicateName
  atgc
  >1_duplicateName
  atgc
  
  Is is possible to run through a large file like this and change the names of only the duplicates?
  >1_uniqueGeneName
  atgc
  >1_anotherUniqueGeneName
  atgc
  >1_duplicateName_1
  atgc
  >1_duplicateName_2
  atgc
  
  
  user298: Sure, this little Perl snippet should do it:
  $ perl -pe 's/$/_$seen{$_}/ if ++$seen{$_}>1 and /^>/; ' file.fa 
  >1_uniqueGeneName
  atgc
  >1_anotherUniqueGeneName
  atgc_2
  >1_duplicateName
  atgc_3
  >1_duplicateName_2
  atgc_4
  
  Or, to make the changes in the original file, use -i:    
  perl -i.bak -pe 's/$/_$seen{$_}/ if ++$seen{$_}>1 and /^>/; ' file.fa 
  
  Note that the first occurrence of a duplicate name isn't changed, the second will become _2, the third _3 etc. 
  Explanation
  
  perl -pe : print each input line after applying the script given by -e to it. 
  ++$seen{$_}>1 : increment the current value stored in the hash %seen for this line ($_) by 1 and compare it to 1. 
  s/$/_$seen{$_}/ if ++$seen{$_}>1 and /^>/ : if the current line starts with a > and the  value stored in the hash %seen for this line is greater than 1 (if this isn't the first time we see this line), replace the end of the line ($) with a _ and the current value in the hash
  
  
  Alternatively, here's the same idea in awk:
  $ awk '(/^>/ && s[$0]++){$0=$0"_"s[$0]}1;' file.fa 
  >1_uniqueGeneName
  atgc
  >1_anotherUniqueGeneName
  atgc
  >1_duplicateName
  atgc
  >1_duplicateName_2
  atgc
  
  To make the changes in the original file (assuming you are using GNU awk which is the default on most Linux versions), use -i inplace:
  awk -iinplace '(/^>/ && s[$0]++){$0=$0"_"s[$0]}1;' file.fa 
  
  Explanation
  In awk, the special variable $0 is the current line. 
  
  (/^>/ && s[$0]++) : if this line starts with a > and incrementing the value stored in the array s for this line by 1 evaluates to true (is greater than 0).
  $0=$0"_"s[$0] :  make the current line be itself with a _ and the value from s appended.
  1; : this is just shorthand for "print this line". If an expression evaluates to true, awk will print the current line. Since 1 is always true, this will print every line. 
  
  
  If you want all of the duplicates to be marked, you need to read the file twice. Once to collect the names and a second to mark them:
  $ awk '{
      if (NR==FNR){
          if(/^>/){
              s[$0]++
          }
          next;
      }
      if(/^>/){
          k[$0]++;
          if(s[$0]>1){
              $0=$0"_"k[$0]
          }
      }
      print
  }' file.fa file.fa
  >1_uniqueGeneName
  atgc
  >1_anotherUniqueGeneName
  atgc
  >1_duplicateName_1
  atgc
  >1_duplicateName_2
  atgc
  
  
  IMPORTANT: note that all of these approaches assume you don't already have sequence names ending with _N where N is a number. If your input file has 2 sequences called foo and one called foo_2, then you will end up with two foo_2:
  $ cat test.fa
  >foo_2
  actg
  >foo
  actg
  >foo
  actg
  $ perl -pe 's/$/_$seen{$_}/ if ++$seen{$_}>1 and /^>/; ' test.fa 
  >foo_2
  actg
  >foo
  actg
  >foo_2
  actg
  
  If this can be an issue for you, use one of the more sophisticated approaches suggested by the other answers.
  
  user77: Sure, using biopython:
  #!/usr/bin/env python
  from Bio import SeqIO
  
  records = set()
  of = open("output.fa", "w")
  for record in SeqIO.parse("foo.fa", "fasta"):
      ID = record.id
      num = 1
      while ID in records:
          ID = "{}_{}".format(record.id, num)
          num += 1
      records.add(ID)
      record.id = ID
      record.name = ID
      record.description = ID
      SeqIO.write(record, of, "fasta")
  of.close()
  
  Change output.fa and foo.fa. One doesn't need to explicitly change the .name and .description, but that's handy to prevent the original ID from not appearing still (after a space).
  Explanation
  
  records = set() This will create a lookup table of all IDs written
  for record in SeqIO.parse("foo.fa", "fasta"): Open foo.fa as a fasta file and iterate over the entries in it. These are objects with an ID, name, description, and sequence attribute (the name and description are the same as the ID if not present).
  ID = record.id Memoize the entry ID (e.g., 1_duplicateName)
  while ID in records: As long as the ID has already been seen, keep looping.
  ID = "{}_{}".format(record.id, num) Start adding increasing numbers after the ID, such as 1_duplicateName_1 and 1_duplicateName_2. This will continue until the ID has not been seen.
  records.add(ID) Add the unseen ID to the set.
  record.id = ID Update the ID, the .name and .description are the same. If you don't do that, then you get output like >1_duplicateName_1 1_duplicateName. That's not really a problem, but it's excessive.
  SeqIO.write(record, of, "fasta") Write the record to the output file in fasta format.
  
  A benefit of biopython is the easy ability to change formats, so instead of fasta one could have substituted Genbank or another format if needed.
  
  user29: And here’s a solution using R (with the Bioconductor):
  fa = ShortRead::readFasta(infile)
  ids = as.character(ShortRead::id(fa))
  fa@id = Biostrings::BStringSet(make.unique(ids, sep = '_'))
  ShortRead::writeFasta(fa, outfile)
  
  
original_text: 982: <p>I have a reference database with contains 100s of sequences in fasta format. Some of these sequences have duplicate names like so:</p>
  
  <pre><code>&gt;1_uniqueGeneName
  atgc
  &gt;1_anotherUniqueGeneName
  atgc
  &gt;1_duplicateName
  atgc
  &gt;1_duplicateName
  atgc
  </code></pre>
  
  <p>Is is possible to run through a large file like this and change the names of only the duplicates?</p>
  
  <pre><code>&gt;1_uniqueGeneName
  atgc
  &gt;1_anotherUniqueGeneName
  atgc
  &gt;1_duplicateName_1
  atgc
  &gt;1_duplicateName_2
  atgc
  </code></pre>
   298: <p>Sure, this little Perl snippet should do it:</p>
  
  <pre><code>$ perl -pe 's/$/_$seen{$_}/ if ++$seen{$_}&gt;1 and /^&gt;/; ' file.fa 
  &gt;1_uniqueGeneName
  atgc
  &gt;1_anotherUniqueGeneName
  atgc_2
  &gt;1_duplicateName
  atgc_3
  &gt;1_duplicateName_2
  atgc_4
  </code></pre>
  
  <p>Or, to make the changes in the original file, use <code>-i</code>:    </p>
  
  <pre><code>perl -i.bak -pe 's/$/_$seen{$_}/ if ++$seen{$_}&gt;1 and /^&gt;/; ' file.fa 
  </code></pre>
  
  <p>Note that the first occurrence of a duplicate name isn't changed, the second will become <code>_2</code>, the third <code>_3</code> etc. </p>
  
  <h3>Explanation</h3>
  
  <ul>
  <li><code>perl -pe</code> : <strong>p</strong>rint each input line after applying the script given by <code>-e</code> to it. </li>
  <li><code>++$seen{$_}&gt;1</code> : increment the current value stored in the hash <code>%seen</code> for this line (<code>$_</code>) by 1 and compare it to <code>1</code>. </li>
  <li><code>s/$/_$seen{$_}/ if ++$seen{$_}&gt;1 and /^&gt;/</code> : if the current line starts with a <code>&gt;</code> and the  value stored in the hash <code>%seen</code> for this line is greater than 1 (if this isn't the first time we see this line), replace the end of the line (<code>$</code>) with a <code>_</code> and the current value in the hash</li>
  </ul>
  
  <hr>
  
  <p>Alternatively, here's the same idea in <code>awk</code>:</p>
  
  <pre><code>$ awk '(/^&gt;/ &amp;&amp; s[$0]++){$0=$0"_"s[$0]}1;' file.fa 
  &gt;1_uniqueGeneName
  atgc
  &gt;1_anotherUniqueGeneName
  atgc
  &gt;1_duplicateName
  atgc
  &gt;1_duplicateName_2
  atgc
  </code></pre>
  
  <p>To make the changes in the original file (assuming you are using GNU <code>awk</code> which is the default on most Linux versions), use <code>-i inplace</code>:</p>
  
  <pre><code>awk -iinplace '(/^&gt;/ &amp;&amp; s[$0]++){$0=$0"_"s[$0]}1;' file.fa 
  </code></pre>
  
  <h3>Explanation</h3>
  
  <p>In awk, the special variable <code>$0</code> is the current line. </p>
  
  <ul>
  <li><code>(/^&gt;/ &amp;&amp; s[$0]++)</code> : if this line starts with a <code>&gt;</code> and incrementing the value stored in the array <code>s</code> for this line by 1 evaluates to true (is greater than 0).</li>
  <li><code>$0=$0"_"s[$0]</code> :  make the current line be itself with a <code>_</code> and the value from <code>s</code> appended.</li>
  <li><code>1;</code> : this is just shorthand for "print this line". If an expression evaluates to true, awk will print the current line. Since <code>1</code> is always true, this will print every line. </li>
  </ul>
  
  <hr>
  
  <p>If you want all of the duplicates to be marked, you need to read the file twice. Once to collect the names and a second to mark them:</p>
  
  <pre><code>$ awk '{
      if (NR==FNR){
          if(/^&gt;/){
              s[$0]++
          }
          next;
      }
      if(/^&gt;/){
          k[$0]++;
          if(s[$0]&gt;1){
              $0=$0"_"k[$0]
          }
      }
      print
  }' file.fa file.fa
  &gt;1_uniqueGeneName
  atgc
  &gt;1_anotherUniqueGeneName
  atgc
  &gt;1_duplicateName_1
  atgc
  &gt;1_duplicateName_2
  atgc
  </code></pre>
  
  <hr>
  
  <p><strong>IMPORTANT</strong>: note that all of these approaches assume you don't already have sequence names ending with <code>_N</code> where <code>N</code> is a number. If your input file has 2 sequences called <code>foo</code> and one called <code>foo_2</code>, then you will end up with two <code>foo_2</code>:</p>
  
  <pre><code>$ cat test.fa
  &gt;foo_2
  actg
  &gt;foo
  actg
  &gt;foo
  actg
  $ perl -pe 's/$/_$seen{$_}/ if ++$seen{$_}&gt;1 and /^&gt;/; ' test.fa 
  &gt;foo_2
  actg
  &gt;foo
  actg
  &gt;foo_2
  actg
  </code></pre>
  
  <p>If this can be an issue for you, use one of the more sophisticated approaches suggested by the other answers.</p>
   77: <p>Sure, using biopython:</p>
  
  <pre><code>#!/usr/bin/env python
  from Bio import SeqIO
  
  records = set()
  of = open("output.fa", "w")
  for record in SeqIO.parse("foo.fa", "fasta"):
      ID = record.id
      num = 1
      while ID in records:
          ID = "{}_{}".format(record.id, num)
          num += 1
      records.add(ID)
      record.id = ID
      record.name = ID
      record.description = ID
      SeqIO.write(record, of, "fasta")
  of.close()
  </code></pre>
  
  <p>Change <code>output.fa</code> and <code>foo.fa</code>. One doesn't need to explicitly change the <code>.name</code> and <code>.description</code>, but that's handy to prevent the original ID from not appearing still (after a space).</p>
  
  <h1>Explanation</h1>
  
  <ul>
  <li><code>records = set()</code> This will create a lookup table of all IDs <strong>written</strong></li>
  <li><code>for record in SeqIO.parse("foo.fa", "fasta"):</code> Open <code>foo.fa</code> as a fasta file and iterate over the <strong>entries</strong> in it. These are objects with an ID, name, description, and sequence attribute (the name and description are the same as the ID if not present).</li>
  <li><code>ID = record.id</code> <a href="https://en.wikipedia.org/wiki/Memoization" rel="nofollow noreferrer">Memoize</a> the entry ID (e.g., <code>1_duplicateName</code>)</li>
  <li><code>while ID in records:</code> As long as the ID has already been seen, keep looping.</li>
  <li><code>ID = "{}_{}".format(record.id, num)</code> Start adding increasing numbers after the ID, such as <code>1_duplicateName_1</code> and <code>1_duplicateName_2</code>. This will continue until the ID has not been seen.</li>
  <li><code>records.add(ID)</code> Add the unseen ID to the set.</li>
  <li><code>record.id = ID</code> Update the ID, the <code>.name</code> and <code>.description</code> are the same. If you don't do that, then you get output like <code>&gt;1_duplicateName_1 1_duplicateName</code>. That's not really a problem, but it's excessive.</li>
  <li><code>SeqIO.write(record, of, "fasta")</code> Write the record to the output file in fasta format.</li>
  </ul>
  
  <p>A benefit of biopython is the easy ability to change formats, so instead of fasta one could have substituted Genbank or another format if needed.</p>
   29: <p>And here’s a solution using R (with the Bioconductor):</p>
  
  <pre class="lang-r prettyprint-override"><code>fa = ShortRead::readFasta(infile)
  ids = as.character(ShortRead::id(fa))
  fa@id = Biostrings::BStringSet(make.unique(ids, sep = '_'))
  ShortRead::writeFasta(fa, outfile)
  </code></pre>
  