text: user7476: I have metabarcoding sequence data (COI) from bulk animal samples (including arthropoda, nematoda, annelida, mollusca) and I want to BLAST all of these sequences. I used following command to do this: blastn -remote -db nt -query COI_all.fasta -num_alignments 2 -out COI_blasted.txt. However this results in errors similar to this post: https://www.biostars.org/p/359971/ . 
  These errors probably appear due to the number of sequences in my file (around 700) and 
  the remote connection is thus interrupted. 
  I found that a solution would be to use blastn with a local database and since the samples are so diverse, I would like to download ALL animal COI sequences from BOLD (or gen bank). It would not be a problem if non-animal (e.g. plant) sequences would also be included.
  I think the BOLD database would be great to BLAST my sequences to. However, I'm currently struggling to find a good way to download all animal COI sequences from BOLD.
  When entering COI-5P as search term on http://v4.boldsystems.org/index.php/Public_SearchTerms I receive error: Your search terms resulted in too many matching terms. Please try again with more specific search criteria.. I could likely download the sequences from all the phyla etc seperately and merge them, but I'd rather just download 1 file.
  I also tried to use the API by running: wget http://v4.boldsystems.org/index.php/API_Public/sequence?marker=COI-5P. A download starts but around 3.7 MB download, it is stuck and the file I receive only contains ~5000 sequences.
  UPDATE: I've contacted BOLD about the stalling behavior and this is their reply: "This issue is because of the large API request that retrieves millions of records, which our system does not handle. Please break up the search by smaller groups, such as classes."
  Does anyone have a solution to download all COI sequences from BOLD in one file?
  I could also download COI sequences from gen bank using the ftp://ftp.ncbi.nlm.nih.gov/blast/db/ URL, but I'm not sure which exact files I need. For 16S, 18S,.. it is obvious, but not for COI. Any suggestions?
  Thanks for the help.
  
  user8629: A straightforward way of getting the sequences from GenBank is from their website using this URL:
  https://www.ncbi.nlm.nih.gov/nuccore?term=COI[Gene Name]
  There you can click on "Send to" -> select "Gene Features" and "FASTA Nucleotide" -> Click "Create File".
  There are >3 mio hits, so the download could take a while.
  
  user8629: I couldn't help but looking a bit deeper into this. There is actually an R package that uses the BOLD API.
  However, they also say that there will be timeouts when downloading large sets of data. Luckily, they suggest a workaround (downloading sequences separately for each taxonomic Class).
  So you should be able to download the marker sequences using the suggested steps:
  # install packages (if necessary)
  install.packages("bold")
  install.packages("taxize")
  install.packages("seqinr")
  # load packages
  library(tidyverse)
  library(bold)    # API interface to BOLD
  library(taxize)  # for NCBI taxonomy lookup
  library(seqinr)  # for FASTA output
  
  # get class-level taxa within "Animalia" from NCBI taxonomy
  taxa <- downstream("Animalia", db = "ncbi", downto = "class")
  # check if taxa present in BOLD
  checks <- bold_tax_name(taxa$Animalia$childtaxa_name)
  taxa_bold <- checks[!is.na(checks$taxon),]$taxon
  
  # download sequences from BOLD for each class-level taxon
  sequences <- map(taxa_bold[1:3], bold_seq, marker = 'COI-5P') %>%
    flatten() %>%
    bind_rows()
  
  # write sequences to file
  write.fasta(
    sequences = as.list(sequences$sequence), 
    names = as.list(sequences$id), 
    nbchar = 80, 
    file.out = 'coi5p.fasta')
  
  Note that I selected the first tree taxa (taxa_bold[1:3]) for testing, so that it runs quicker. If you want to get all taxa, just remove the [1:3].
  One last importand caveat: In the help for bold_seq it says that
  
  Notes from BOLD on the marker param:  "All markers for a specimen matching the search stringwill be returned. ie. A record with COI-5P and ITS will return sequence data for both markers evenif only COI-5P was specified."You will likely end up with data with markers that you did not request - just be sure to filter thoseout as needed.
  
  So you might want to double-check that the downloaded sequences are actually COI-5P. But if you are using them as a BLAST database, you might also be happy with COI-5P and ITS markers being present in the database (it affects the e-values, though).
  
original_text: 7476: <p>I have metabarcoding sequence data (COI) from bulk animal samples (including arthropoda, nematoda, annelida, mollusca) and I want to BLAST all of these sequences. I used following command to do this: <code>blastn -remote -db nt -query COI_all.fasta -num_alignments 2 -out COI_blasted.txt</code>. However this results in errors similar to this post: <a href="https://www.biostars.org/p/359971/" rel="nofollow noreferrer">https://www.biostars.org/p/359971/</a> . </p>
  
  <p>These errors probably appear due to the number of sequences in my file (around 700) and 
  the remote connection is thus interrupted. </p>
  
  <p>I found that a solution would be to use <code>blastn</code> with a local database and since the samples are so diverse, I would like to download ALL animal COI sequences from BOLD (or gen bank). It would not be a problem if non-animal (e.g. plant) sequences would also be included.</p>
  
  <p>I think the BOLD database would be great to BLAST my sequences to. However, I'm currently struggling to find a good way to download all animal <code>COI</code> sequences from BOLD.</p>
  
  <p>When entering <code>COI-5P</code> as search term on <a href="http://v4.boldsystems.org/index.php/Public_SearchTerms" rel="nofollow noreferrer">http://v4.boldsystems.org/index.php/Public_SearchTerms</a> I receive error: <code>Your search terms resulted in too many matching terms. Please try again with more specific search criteria.</code>. I could likely download the sequences from all the <code>phyla</code> etc seperately and merge them, but I'd rather just download 1 file.</p>
  
  <p>I also tried to use the API by running: <code>wget http://v4.boldsystems.org/index.php/API_Public/sequence?marker=COI-5P</code>. A download starts but around 3.7 MB download, it is stuck and the file I receive only contains ~5000 sequences.</p>
  
  <p><strong>UPDATE: I've contacted BOLD about the stalling behavior and this is their reply: "This issue is because of the large API request that retrieves millions of records, which our system does not handle. Please break up the search by smaller groups, such as classes."</strong></p>
  
  <p><strong>Does anyone have a solution to download all COI sequences from BOLD in one file?</strong></p>
  
  <p>I could also download COI sequences from gen bank using the <a href="ftp://ftp.ncbi.nlm.nih.gov/blast/db/" rel="nofollow noreferrer">ftp://ftp.ncbi.nlm.nih.gov/blast/db/</a> URL, but I'm not sure which exact files I need. For <code>16S, 18S,..</code> it is obvious, but not for COI. Any suggestions?</p>
  
  <p>Thanks for the help.</p>
   8629: <p>A straightforward way of getting the sequences from GenBank is from their website using this URL:
  <a href="https://www.ncbi.nlm.nih.gov/nuccore?term=COI%5BGene%20Name%5D" rel="nofollow noreferrer">https://www.ncbi.nlm.nih.gov/nuccore?term=COI[Gene Name]</a></p>
  
  <p>There you can click on "Send to" -> select "Gene Features" and "FASTA Nucleotide" -> Click "Create File".</p>
  
  <p>There are >3 mio hits, so the download could take a while.</p>
   8629: <p>I couldn't help but looking a bit deeper into this. There is actually an <a href="https://cran.r-project.org/web/packages/bold/" rel="nofollow noreferrer">R package that uses the BOLD API</a>.</p>
  
  <p>However, they also say that there will be timeouts when downloading large sets of data. Luckily, they suggest a <a href="https://docs.ropensci.org/bold/#large-data" rel="nofollow noreferrer">workaround</a> (downloading sequences separately for each taxonomic Class).</p>
  
  <p>So you should be able to download the marker sequences using the suggested steps:</p>
  
  <pre><code># install packages (if necessary)
  install.packages("bold")
  install.packages("taxize")
  install.packages("seqinr")
  # load packages
  library(tidyverse)
  library(bold)    # API interface to BOLD
  library(taxize)  # for NCBI taxonomy lookup
  library(seqinr)  # for FASTA output
  
  # get class-level taxa within "Animalia" from NCBI taxonomy
  taxa &lt;- downstream("Animalia", db = "ncbi", downto = "class")
  # check if taxa present in BOLD
  checks &lt;- bold_tax_name(taxa<span class="math-container">$Animalia$</span>childtaxa_name)
  taxa_bold &lt;- checks[!is.na(checks<span class="math-container">$taxon),]$</span>taxon
  
  # download sequences from BOLD for each class-level taxon
  sequences &lt;- map(taxa_bold[1:3], bold_seq, marker = 'COI-5P') %&gt;%
    flatten() %&gt;%
    bind_rows()
  
  # write sequences to file
  write.fasta(
    sequences = as.list(sequences<span class="math-container">$sequence), 
    names = as.list(sequences$</span>id), 
    nbchar = 80, 
    file.out = 'coi5p.fasta')
  </code></pre>
  
  <p>Note that I selected the first tree taxa (<code>taxa_bold[1:3]</code>) for testing, so that it runs quicker. If you want to get all taxa, just remove the <code>[1:3]</code>.</p>
  
  <p>One last importand caveat: In the <a href="https://cran.r-project.org/web/packages/bold/bold.pdf#Rfn.bold.Rul.seq" rel="nofollow noreferrer">help for <code>bold_seq</code></a> it says that</p>
  
  <blockquote>
    <p>Notes from BOLD on the <code>marker</code> param:  "All markers for a specimen matching the search stringwill be returned. ie. A record with COI-5P and ITS will return sequence data for both markers evenif only COI-5P was specified."You will likely end up with data with markers that you did not request - just be sure to filter thoseout as needed.</p>
  </blockquote>
  
  <p>So you might want to double-check that the downloaded sequences are actually COI-5P. But if you are using them as a BLAST database, you might also be happy with COI-5P <strong>and</strong> ITS markers being present in the database (it affects the e-values, though).</p>
  