text: user57: I am trying to build a workflow that gets data automatically from databases of sequencing reads (SRA - sequence read archive, ENA - european nucleotide archive). Till now I was pulling everything from SRA and my script have worked, but now I encountered a first entry that is only in ENA (ERR2135453), not in SRA. And I found that my download script has its limitations.
  My bash script looks like this (requires aspera ascp and fastq-dump) and I wanted to make changes that are commented now
  #!/bin/bash
  
  ACCESION=$1
  OPENSSH=/my/part/to/key/asperaweb_id_dsa.openssh
  
  # if [ ${ACCESION:0:3} == SRA ]; then
      URL=anonftp@ftp-private.ncbi.nlm.nih.gov:/sra/sra-instant/reads/ByRun/sra/${ACCESION::3}/${ACCESION::6}/"$ACCESION"/"$ACCESION".sra data/
  # else
  #    URL=ena_url
  # fi    
  
  ascp -QT -i $OPENSSH $URL
  
  fastq-dump data/"$ACCESION".sra --outdir data/ --split-files --gzip
  rm data/"$ACCESION".sra
  
  The problem is in getting ENA url. Unlike SRA url that I can easily get from the accession (anonftp@ftp-private.ncbi.nlm.nih.gov:/sra/sra-instant/reads/ByRun/sra/${ACCESION::3}/${ACCESION::6}/"$ACCESION"/"$ACCESION".sra data/), the ftp server of ENA is bit more confusing. The url corresponding to ERR2135453 accession is ftp://ftp.sra.ebi.ac.uk//vol1/err/ERR213/003/ERR2135453. Where 003 there comes from? Also not all reads are in err subdirectory. 
  Is there an easy way how to download reads from ENA knowing just the accession number?
  --- edit ---
  I found on the relevant ENA site about downloading data using aspera that ftp urls should be possible to derive only from accession also in ENA.
  
  Submitted read data files are organised by submission accession number
    under vol1/ directory in ftp.sra.ebi.ac.uk:
  ftp://ftp.sra.ebi.ac.uk/vol1//
  where  contains the first 6 letters and
    numbers of the SRA Submission accession.
  For example, the files submitted in the SRA Submission ERA007448 are
    available at: ftp://ftp.sra.ebi.ac.uk/vol1/ERA007/ERA007448/.
  
  The reason why there is this strange 003 will probably be that it's quite recent submission and it has not been properly categorized yet (other accessions I tried were on "expected" urls).
  However, the problem still persist. Is there a way how to fetch data of new submissions using accessions only?
  
  user57: Apparently there is no 100% reliable solution, but in the most cases, you should be able to use the enaBrowserTools python scripts (enaDataGet) to quite reliably download data.
  The run ERR2135453 is a special case because the sample submission (ERS1939590) for it has been cancelled. The solution for these reads is to download the data manually from the ftp location.
  This answer is based on kind assistance of Suran from ebi helpdesk.
  
  user235: Here are two python functions I use for getting stuff from ENA based on an accession:
  
  import os
  import itertools
  from urllib.request import urlopen
  
  def fetch_ENA_files(accession):
      """Get the names of the files matching the ENA accession"""
  
      # Get the paths to the files
      url = "http://www.ebi.ac.uk/ena/data/warehouse/filereport?accession=%s&result=read_run&fields=fastq_ftp" \
            % accession
  
      try:
          paths = urlopen(url).readlines()[1:]
      except:
          E.debug("couldn't access %s" % url)
          raise
  
      paths = list(itertools.chain.from_iterable(
          [p.strip().split(";") for p in paths]))
      filenames = [os.path.basename(x) for x in paths]
      dl_paths = [os.path.join(*x.split("/")[1:]) for x in paths]
      return filenames, dl_paths
  
  def fetch_ENA(dl_path, outdir, protocol="ascp"):
      """Fetch fastq from ENA given accession"""
  
      if protocol == "ascp":
          statement = """ascp -QT -P33001 -l 100M -i $ASCP_KEY_PATH
                         era-fasp@fasp.sra.ebi.ac.uk:/%(dl_path)s %(outdir)s """ % locals()
      elif protocol == "http":
          fn = os.path.basename(dl_path)
          outfile = os.path.join(outdir, fn)
          statement = "wget -O %(outfile)s ftp://ftp.sra.ebi.ac.uk:/%(dl_path)s" % locals()
  
      return statement
  
  fetch_ENA_files queries the ENA database for the names of all the files associated with a accession and returns a tuple of lists. The first entry in the tuple contains the filenames (which you might need later in your application), the second contains a list of paths on the ENA servers where those files can be found. 
  fetch_ENA(dl_path, outdir, protocol) takes a download path and downloads it to outdir using with ftp or ascp.
  You might use it like:
  filenames, paths = fetch_ENA_files("ERR2135453")
  for dlp in paths:
      bash_statement = fetch_ENA(dlp, "downloads", "ascp")
      os.system(bash_statement)
  
  (we don't we actaully use our modules to send it to the cluster)
  
original_text: 57: <p>I am trying to build a workflow that gets data automatically from databases of sequencing reads (SRA - sequence read archive, ENA - european nucleotide archive). Till now I was pulling everything from SRA and my script have worked, but now I encountered a first entry that is only in ENA (<a href="https://www.ebi.ac.uk/ena/data/view/ERR2135453" rel="nofollow noreferrer">ERR2135453</a>), <a href="https://www.ncbi.nlm.nih.gov/sra/?term=ERR2135453" rel="nofollow noreferrer">not in SRA</a>. And I found that my download script has its limitations.</p>
  
  <p>My bash script looks like this (requires aspera <code>ascp</code> and <code>fastq-dump</code>) and I wanted to make changes that are commented now</p>
  
  <pre><code>#!/bin/bash
  
  ACCESION=$1
  OPENSSH=/my/part/to/key/asperaweb_id_dsa.openssh
  
  # if [ ${ACCESION:0:3} == SRA ]; then
      URL=anonftp@ftp-private.ncbi.nlm.nih.gov:/sra/sra-instant/reads/ByRun/sra/${ACCESION::3}/${ACCESION::6}/"$ACCESION"/"$ACCESION".sra data/
  # else
  #    URL=ena_url
  # fi    
  
  ascp -QT -i $OPENSSH $URL
  
  fastq-dump data/"$ACCESION".sra --outdir data/ --split-files --gzip
  rm data/"$ACCESION".sra
  </code></pre>
  
  <p>The problem is in getting ENA url. Unlike SRA url that I can easily get from the accession (<code>anonftp@ftp-private.ncbi.nlm.nih.gov:/sra/sra-instant/reads/ByRun/sra/${ACCESION::3}/${ACCESION::6}/"$ACCESION"/"$ACCESION".sra data/</code>), the ftp server of ENA is bit more confusing. The url corresponding to ERR2135453 accession is <code>ftp://ftp.sra.ebi.ac.uk//vol1/err/ERR213/003/ERR2135453</code>. Where <code>003</code> there comes from? Also not all reads are in <code>err</code> subdirectory. </p>
  
  <p>Is there an easy way how to download reads from ENA knowing just the accession number?</p>
  
  <p>--- edit ---</p>
  
  <p>I found on the relevant ENA <a href="https://www.ebi.ac.uk/ena/browse/read-download#downloading_files_aspera" rel="nofollow noreferrer">site about downloading data using aspera</a> that ftp urls should be possible to derive only from accession also in ENA.</p>
  
  <blockquote>
    <p>Submitted read data files are organised by submission accession number
    under vol1/ directory in ftp.sra.ebi.ac.uk:</p>
    
    <p><a href="ftp://ftp.sra.ebi.ac.uk/vol1/" rel="nofollow noreferrer">ftp://ftp.sra.ebi.ac.uk/vol1/</a>/</p>
    
    <p>where  contains the first 6 letters and
    numbers of the SRA Submission accession.</p>
    
    <p>For example, the files submitted in the SRA Submission ERA007448 are
    available at: <a href="ftp://ftp.sra.ebi.ac.uk/vol1/ERA007/ERA007448/" rel="nofollow noreferrer">ftp://ftp.sra.ebi.ac.uk/vol1/ERA007/ERA007448/</a>.</p>
  </blockquote>
  
  <p>The reason why there is this strange <code>003</code> will probably be that it's quite recent submission and it has not been properly categorized yet (other accessions I tried were on "expected" urls).</p>
  
  <p>However, the problem still persist. Is there a way how to fetch data of new submissions using accessions only?</p>
   57: <p>Apparently there is no <strong>100% reliable</strong> solution, but in the most cases, you should be able to use the <a href="https://github.com/enasequence/enaBrowserTools" rel="nofollow noreferrer">enaBrowserTools</a> python scripts (enaDataGet) to quite reliably download data.</p>
  
  <p>The run ERR2135453 is a special case because the sample submission (ERS1939590) for it has been cancelled. The solution for these reads is to download the data manually from the ftp location.</p>
  
  <p>This answer is based on kind assistance of Suran from ebi helpdesk.</p>
   235: <p>Here are two python functions I use for getting stuff from ENA based on an accession:
  </p>
  
  <pre class="lang-py prettyprint-override"><code>import os
  import itertools
  from urllib.request import urlopen
  
  def fetch_ENA_files(accession):
      """Get the names of the files matching the ENA accession"""
  
      # Get the paths to the files
      url = "http://www.ebi.ac.uk/ena/data/warehouse/filereport?accession=%s&amp;result=read_run&amp;fields=fastq_ftp" \
            % accession
  
      try:
          paths = urlopen(url).readlines()[1:]
      except:
          E.debug("couldn't access %s" % url)
          raise
  
      paths = list(itertools.chain.from_iterable(
          [p.strip().split(";") for p in paths]))
      filenames = [os.path.basename(x) for x in paths]
      dl_paths = [os.path.join(*x.split("/")[1:]) for x in paths]
      return filenames, dl_paths
  
  def fetch_ENA(dl_path, outdir, protocol="ascp"):
      """Fetch fastq from ENA given accession"""
  
      if protocol == "ascp":
          statement = """ascp -QT -P33001 -l 100M -i $ASCP_KEY_PATH
                         era-fasp@fasp.sra.ebi.ac.uk:/%(dl_path)s %(outdir)s """ % locals()
      elif protocol == "http":
          fn = os.path.basename(dl_path)
          outfile = os.path.join(outdir, fn)
          statement = "wget -O %(outfile)s ftp://ftp.sra.ebi.ac.uk:/%(dl_path)s" % locals()
  
      return statement
  </code></pre>
  
  <p><code>fetch_ENA_files</code> queries the ENA database for the names of all the files associated with a accession and returns a tuple of lists. The first entry in the tuple contains the filenames (which you might need later in your application), the second contains a list of paths on the ENA servers where those files can be found. </p>
  
  <p><code>fetch_ENA(dl_path, outdir, protocol)</code> takes a download path and downloads it to <code>outdir</code> using with ftp or ascp.</p>
  
  <p>You might use it like:</p>
  
  <pre class="lang-py prettyprint-override"><code>filenames, paths = fetch_ENA_files("ERR2135453")
  for dlp in paths:
      bash_statement = fetch_ENA(dlp, "downloads", "ascp")
      os.system(bash_statement)
  </code></pre>
  
  <p>(we don't we actaully use our modules to send it to the cluster)</p>
  