text: user3686: I am trying to download multiple SRA files located in several SRA accessions. Some of my accession numbers are as follows:
  SRR1002678
  SRR1158898
  SRR1158907
  SRR1158950
  SRR1181152
  SRR1181155
  SRR1181196
  SRR1181204
  SRR1181205
  SRR1181208
  SRR1269497
  SRR2100227
  SRR2100419
  SRR2100830
  SRR2101344
  SRR671725
  SRR671726
  SRR671740
  SRR671759
  SRR671760
  SRR671761
  SRR671779
  SRR671782
  SRR671815
  SRR671821
  SRR671842
  SRR671862
  SRR671863
  SRR671864
  SRR671865
  .............
  
  I tried the following command:
  for i in {SRR1002678..SRR1184123}; do wget ftp://ftp.ncbi.nlm.nih.gov/sra/sra-instant/reads/ByRun/sra/SRR/SRR1*/$i; done
  
  but this did not work.
  I tried prefetch as well:
  sort -u SRR.txt | parallel -j 4 "prefetch {}"
  
  where SRR.txt is the list of my SRA IDs. However, with this command only a few files are downloaded as *.sra files, while with several other SRA IDs I get the following error:
  2018-11-15T06:17:20 prefetch.2.8.2: 1) Downloading 'SRR1183962'...
  2018-11-15T06:17:20 prefetch.2.8.2:  Downloading via https...
  2018-11-15T06:20:21 prefetch.2.8.2: 1) 'SRR1183962' was downloaded successfully
  2018-11-15T06:20:27 prefetch.2.8.2: 'SRR1183962' has 0 unresolved dependencies
  2018-11-15T06:20:38 prefetch.2.8.2: 'SRR1183962' has no remote vdbcache
  
  2018-11-15T06:18:59 prefetch.2.8.2: 1) Downloading 'SRR1184034'...
  2018-11-15T06:18:59 prefetch.2.8.2:  Downloading via https...
  2018-11-15T06:20:40 prefetch.2.8.2: 1) failed to download SRR1184034
  2018-11-15T06:20:40 prefetch.2.8.2 sys: libs/kns/unix/syssock.c:606:KSocketTimedRead: timeout exhausted while reading file within network system module - mbedtls_ssl_read returned -76 ( NET - Reading information from the socket failed )
  2018-11-15T06:20:40 prefetch.2.8.2 int: libs/kns/unix/syssock.c:606:KSocketTimedRead: timeout exhausted while reading file within network system module -
  
  I do not have a range of SRA files to download but quite random SRA files to download. What am I doing wrong? Any help is highly appreciated.
  
  user298: First of all, the brace expansion you tried to use isn't valid. While bash can expand {1..3} to 1 2 3, it has no way of knowing how to expand {SRR1002678..SRR1184123}:
  $ echo {1..3}
  1 2 3
  $ echo {SRR1002678..SRR1184123}
  {SRR1002678..SRR1184123}
  
  Perhaps you meant SRR{1002678..1184123}? But there's no need for this, you already have the accessions in a file and can read that. I see that the FTP server you are accessing has organized the data like this:
  sra/sra-instant/reads/ByRun/sra/SRR/SRRNNN/SRRNNNXXXX/SRRNNNXXXX.sra
  
  Where NNN are the 1st three digits of the accession. So, for example, the file for accession SRR1002678 is at:
  ftp://ftp.ncbi.nlm.nih.gov/sra/sra-instant/reads/ByRun/sra/SRR/SRR118/SRR1181152/SRR1181152.sra
  
  With this in mind, you can write a little script that extracts the URL from each accession:
  while read i; do 
      n=$(echo "$i" | sed -E 's/(SRR...).*/\1/'); 
      wget ftp://ftp.ncbi.nlm.nih.gov/sra/sra-instant/reads/ByRun/sra/SRR/$n/$i/$i.sra
  done < SRR.txt
  
  The only trick is the sed command which simply looks for SRR and the three next characters and removes everything else on the line (it "captures" the found pattern and then replaces the line with what was captured (\1)).
  
original_text: 3686: <p>I am trying to download multiple SRA files located in several SRA accessions. Some of my accession numbers are as follows:</p>
  
  <pre><code>SRR1002678
  SRR1158898
  SRR1158907
  SRR1158950
  SRR1181152
  SRR1181155
  SRR1181196
  SRR1181204
  SRR1181205
  SRR1181208
  SRR1269497
  SRR2100227
  SRR2100419
  SRR2100830
  SRR2101344
  SRR671725
  SRR671726
  SRR671740
  SRR671759
  SRR671760
  SRR671761
  SRR671779
  SRR671782
  SRR671815
  SRR671821
  SRR671842
  SRR671862
  SRR671863
  SRR671864
  SRR671865
  .............
  </code></pre>
  
  <p>I tried the following command:</p>
  
  <pre><code>for i in {SRR1002678..SRR1184123}; do wget ftp://ftp.ncbi.nlm.nih.gov/sra/sra-instant/reads/ByRun/sra/SRR/SRR1*/$i; done
  </code></pre>
  
  <p>but this did not work.</p>
  
  <p>I tried <code>prefetch</code> as well:</p>
  
  <pre><code>sort -u SRR.txt | parallel -j 4 "prefetch {}"
  </code></pre>
  
  <p>where SRR.txt is the list of my SRA IDs. However, with this command only a few files are downloaded as *.sra files, while with several other SRA IDs I get the following error:</p>
  
  <pre><code>2018-11-15T06:17:20 prefetch.2.8.2: 1) Downloading 'SRR1183962'...
  2018-11-15T06:17:20 prefetch.2.8.2:  Downloading via https...
  2018-11-15T06:20:21 prefetch.2.8.2: 1) 'SRR1183962' was downloaded successfully
  2018-11-15T06:20:27 prefetch.2.8.2: 'SRR1183962' has 0 unresolved dependencies
  2018-11-15T06:20:38 prefetch.2.8.2: 'SRR1183962' has no remote vdbcache
  
  2018-11-15T06:18:59 prefetch.2.8.2: 1) Downloading 'SRR1184034'...
  2018-11-15T06:18:59 prefetch.2.8.2:  Downloading via https...
  2018-11-15T06:20:40 prefetch.2.8.2: 1) failed to download SRR1184034
  2018-11-15T06:20:40 prefetch.2.8.2 sys: libs/kns/unix/syssock.c:606:KSocketTimedRead: timeout exhausted while reading file within network system module - mbedtls_ssl_read returned -76 ( NET - Reading information from the socket failed )
  2018-11-15T06:20:40 prefetch.2.8.2 int: libs/kns/unix/syssock.c:606:KSocketTimedRead: timeout exhausted while reading file within network system module -
  </code></pre>
  
  <p>I do not have a range of SRA files to download but quite random SRA files to download. What am I doing wrong? Any help is highly appreciated.</p>
   298: <p>First of all, the brace expansion you tried to use isn't valid. While bash can expand <code>{1..3}</code> to <code>1 2 3</code>, it has no way of knowing how to expand <code>{SRR1002678..SRR1184123}</code>:</p>
  
  <pre><code><span class="math-container">$ echo {1..3}
  1 2 3
  $</span> echo {SRR1002678..SRR1184123}
  {SRR1002678..SRR1184123}
  </code></pre>
  
  <p>Perhaps you meant <code>SRR{1002678..1184123}</code>? But there's no need for this, you already have the accessions in a file and can read that. I see that the FTP server you are accessing has organized the data like this:</p>
  
  <pre><code>sra/sra-instant/reads/ByRun/sra/SRR/SRRNNN/SRRNNNXXXX/SRRNNNXXXX.sra
  </code></pre>
  
  <p>Where <code>NNN</code> are the 1st three digits of the accession. So, for example, the file for accession <code>SRR1002678</code> is at:</p>
  
  <pre><code>ftp://ftp.ncbi.nlm.nih.gov/sra/sra-instant/reads/ByRun/sra/SRR/SRR118/SRR1181152/SRR1181152.sra
  </code></pre>
  
  <p>With this in mind, you can write a little script that extracts the URL from each accession:</p>
  
  <pre><code>while read i; do 
      n=<span class="math-container">$(echo "$</span>i" | sed -E 's/(SRR...).*/\1/'); 
      wget ftp://ftp.ncbi.nlm.nih.gov/sra/sra-instant/reads/ByRun/sra/SRR/<span class="math-container">$n/$</span>i/$i.sra
  done &lt; SRR.txt
  </code></pre>
  
  <p>The only trick is the <code>sed</code> command which simply looks for <code>SRR</code> and the three next characters and removes everything else on the line (it "captures" the found pattern and then replaces the line with what was captured (<code>\1</code>)).</p>
  