text: user9868: I am using the following command:
  from Bio import SeqIO
  import sys
  import re 
  
  
  fasta_file = (sys.argv[1])
  for myfile in SeqIO.parse(fasta_file, "fasta"):
  if len(myfile.seq) > 250:
      gene_id = myfile.id
      mylist = re.match(r"H149xcV_[^\W_]+_[^\W_]+_[^\W_])_[^\W_]+", gene_id)
      print (">"+mylist.group(1)) 
  
  And this is providing me with duplicates of the same gene:
  >H149xcV_Fge342_r3_h2
  >H149xcV_bTr423_r3_h2
  >H149xcV_kN893_r3_h2
  >H149xcV_DNp021_r3_h2
  >H149xcV_JEP3324_r3_h2
  >H149xcV_JEP3324_r3_h2
  >H149xcV_JEP3324_r3_h2
  >H149xcV_SRt424234_r3_h2
  >H149xcV_Fge342_r3_h2
  >H149xcV_Fge342_r3_h2
  
  How can I reformat my command so that I only receive unique gene id's:
  >H149xcV_Fge342_r3_h2
  >H149xcV_bTr423_r3_h2
  >H149xcV_kN893_r3_h2
  >H149xcV_DNp021_r3_h2
  >H149xcV_JEP3324_r3_h2
  >H149xcV_SRt424234_r3_h2
  
  
  user10049: from Bio import SeqIO
  import sys
  import re 
  
  unique = []
  fasta_file = (sys.argv[1])
  for myfile in SeqIO.parse(fasta_file, "fasta"):
      if len(myfile) > 250:
          gene_id = myfile.id
          if gene.id not in unique:
              unique.append(gene.id)
              mylist = re.match(r"H149xcV_[^\W_]+_[^\W_]+_[^\W_])_[^\W_]+", gene_id)
              print (">"+mylist.group(1))
  
  
  user10367: Perhaps itertools might be an option? From the itertools page / moreitertools project
  https://docs.python.org/3/library/itertools.html#itertools-recipes
  def unique_everseen(iterable, key=None):
      "List unique elements, preserving order. Remember all elements ever seen."
      # unique_everseen('AAAABBBCCDAABBB') --> A B C D
      # unique_everseen('ABBCcAD', str.lower) --> A B C D
      seen = set()
      seen_add = seen.add
      if key is None:
          for element in filterfalse(seen.__contains__, iterable):
              seen_add(element)
              yield element
      else:
          for element in iterable:
              k = key(element)
              if k not in seen:
                  seen_add(k)
                  yield element
  
  
  user388: I would use a dict so that the lookup time is constant. If you are creating an array with all the unique gene IDs, then checking if a gene is in that array, your runtime is O(N2). When you create an entry in the dict, the time to access is constant, and if the entry is already present, it will just overwrite it. Something like this should do it for you:
  from Bio import SeqIO
      
  genes = {}
  fasta_file = sys.argv[1]
  for entry in SeqIO.parse( fasta_file, "fasta" ):
      genes[ entry.id ] = entry.seq
      
  
  Then you can write a new, unique fasta file:
  with open('unique_genes.fasta', 'w') as out:
      for key,value in genes.items():
          out.write( '>%s\n%s\n'%(key,value) )
  
  
  user298: I wouldn't use Python for this, myself. Instead you can use the FastaToTbl and TblToFasta scripts I have posted before, and pipe to standard *nix utilities:
  FastaToTbl file.fa | sort -u | TblToFasta > file.uniq.fa
  
  Or, if you don't have GNU sort:
  FastaToTbl file.fa | sort | uniq | TblToFasta > file.uniq.fa
  
  Alternatively, you can also do something like this:
  awk -v RS='>' -v ORS='>' '++a[$0]==1' file.fa | sed 's/>$//'
  
  For example, given this input file:
  $ cat file.fa
  >seq1
  ACTTCGCAGAGGCTTCGGAGAGA
  ACTTCGCAGAGGCTTCGGAGAGA
  ACTTCGCAGAGGCTTCGGAGAGA
  >seq2
  ATGGCGCGCTTAGGAGCGCTAGGACT
  >seq2
  ATGGCGCGCTTAGGAGCGCTAGGACT
  >seq1
  ACTTCGCAGAGGCTTCGGAGAGA
  ACTTCGCAGAGGCTTCGGAGAGA
  ACTTCGCAGAGGCTTCGGAGAGA
  >seq1
  ACTTCGCAGAGGCTTCGGAGAGA
  ACTTCGCAGAGGCTTCGGAGAGA
  ACTTCGCAGAGGCTTCGGAGAGA
  
  You can do:
  $ awk -v RS='>' -v ORS='>' '++a[$0]==1' file.fa | sed 's/>$//'
  >seq1
  ACTTCGCAGAGGCTTCGGAGAGA
  ACTTCGCAGAGGCTTCGGAGAGA
  ACTTCGCAGAGGCTTCGGAGAGA
  >seq2
  ATGGCGCGCTTAGGAGCGCTAGGACT
  
  Both of the above approaches have the advantage of not needing to store the entire file in memory.
  
original_text: 9868: <p>I am using the following command:</p>
  <pre><code>from Bio import SeqIO
  import sys
  import re 
  
  
  fasta_file = (sys.argv[1])
  for myfile in SeqIO.parse(fasta_file, &quot;fasta&quot;):
  if len(myfile.seq) &gt; 250:
      gene_id = myfile.id
      mylist = re.match(r&quot;H149xcV_[^\W_]+_[^\W_]+_[^\W_])_[^\W_]+&quot;, gene_id)
      print (&quot;&gt;&quot;+mylist.group(1)) 
  </code></pre>
  <p>And this is providing me with duplicates of the same gene:</p>
  <pre><code>&gt;H149xcV_Fge342_r3_h2
  &gt;H149xcV_bTr423_r3_h2
  &gt;H149xcV_kN893_r3_h2
  &gt;H149xcV_DNp021_r3_h2
  &gt;H149xcV_JEP3324_r3_h2
  &gt;H149xcV_JEP3324_r3_h2
  &gt;H149xcV_JEP3324_r3_h2
  &gt;H149xcV_SRt424234_r3_h2
  &gt;H149xcV_Fge342_r3_h2
  &gt;H149xcV_Fge342_r3_h2
  </code></pre>
  <p>How can I reformat my command so that I only receive unique gene id's:</p>
  <pre><code>&gt;H149xcV_Fge342_r3_h2
  &gt;H149xcV_bTr423_r3_h2
  &gt;H149xcV_kN893_r3_h2
  &gt;H149xcV_DNp021_r3_h2
  &gt;H149xcV_JEP3324_r3_h2
  &gt;H149xcV_SRt424234_r3_h2
  </code></pre>
   10049: <pre><code>from Bio import SeqIO
  import sys
  import re 
  
  unique = []
  fasta_file = (sys.argv[1])
  for myfile in SeqIO.parse(fasta_file, &quot;fasta&quot;):
      if len(myfile) &gt; 250:
          gene_id = myfile.id
          if gene.id not in unique:
              unique.append(gene.id)
              mylist = re.match(r&quot;H149xcV_[^\W_]+_[^\W_]+_[^\W_])_[^\W_]+&quot;, gene_id)
              print (&quot;&gt;&quot;+mylist.group(1))
  </code></pre>
   10367: <p>Perhaps itertools might be an option? From the itertools page / moreitertools project
  <a href="https://docs.python.org/3/library/itertools.html#itertools-recipes" rel="nofollow noreferrer">https://docs.python.org/3/library/itertools.html#itertools-recipes</a></p>
  <pre class="lang-py prettyprint-override"><code>def unique_everseen(iterable, key=None):
      &quot;List unique elements, preserving order. Remember all elements ever seen.&quot;
      # unique_everseen('AAAABBBCCDAABBB') --&gt; A B C D
      # unique_everseen('ABBCcAD', str.lower) --&gt; A B C D
      seen = set()
      seen_add = seen.add
      if key is None:
          for element in filterfalse(seen.__contains__, iterable):
              seen_add(element)
              yield element
      else:
          for element in iterable:
              k = key(element)
              if k not in seen:
                  seen_add(k)
                  yield element
  </code></pre>
   388: <p>I would use a <code>dict</code> so that the lookup time is constant. If you are creating an array with all the unique gene IDs, then checking if a gene is in that array, your runtime is O(N<sup>2</sup>). When you create an entry in the <code>dict</code>, the time to access is constant, and if the entry is already present, it will just overwrite it. Something like this should do it for you:</p>
  <pre><code>from Bio import SeqIO
      
  genes = {}
  fasta_file = sys.argv[1]
  for entry in SeqIO.parse( fasta_file, &quot;fasta&quot; ):
      genes[ entry.id ] = entry.seq
      
  </code></pre>
  <p>Then you can write a new, unique fasta file:</p>
  <pre><code>with open('unique_genes.fasta', 'w') as out:
      for key,value in genes.items():
          out.write( '&gt;%s\n%s\n'%(key,value) )
  </code></pre>
   298: <p>I wouldn't use Python for this, myself. Instead you can use the FastaToTbl and TblToFasta scripts <a href="https://bioinformatics.stackexchange.com/a/2651/298">I have posted before</a>, and pipe to standard *nix utilities:</p>
  <pre><code>FastaToTbl file.fa | sort -u | TblToFasta &gt; file.uniq.fa
  </code></pre>
  <p>Or, if you don't have GNU <code>sort</code>:</p>
  <pre><code>FastaToTbl file.fa | sort | uniq | TblToFasta &gt; file.uniq.fa
  </code></pre>
  <p>Alternatively, you can also do something like this:</p>
  <pre><code>awk -v RS='&gt;' -v ORS='&gt;' '++a[$0]==1' file.fa | sed 's/&gt;$//'
  </code></pre>
  <p>For example, given this input file:</p>
  <pre><code>$ cat file.fa
  &gt;seq1
  ACTTCGCAGAGGCTTCGGAGAGA
  ACTTCGCAGAGGCTTCGGAGAGA
  ACTTCGCAGAGGCTTCGGAGAGA
  &gt;seq2
  ATGGCGCGCTTAGGAGCGCTAGGACT
  &gt;seq2
  ATGGCGCGCTTAGGAGCGCTAGGACT
  &gt;seq1
  ACTTCGCAGAGGCTTCGGAGAGA
  ACTTCGCAGAGGCTTCGGAGAGA
  ACTTCGCAGAGGCTTCGGAGAGA
  &gt;seq1
  ACTTCGCAGAGGCTTCGGAGAGA
  ACTTCGCAGAGGCTTCGGAGAGA
  ACTTCGCAGAGGCTTCGGAGAGA
  </code></pre>
  <p>You can do:</p>
  <pre><code>$ awk -v RS='&gt;' -v ORS='&gt;' '++a[$0]==1' file.fa | sed 's/&gt;$//'
  &gt;seq1
  ACTTCGCAGAGGCTTCGGAGAGA
  ACTTCGCAGAGGCTTCGGAGAGA
  ACTTCGCAGAGGCTTCGGAGAGA
  &gt;seq2
  ATGGCGCGCTTAGGAGCGCTAGGACT
  </code></pre>
  <p>Both of the above approaches have the advantage of not needing to store the entire file in memory.</p>
  