text: user3235: I wish to extract metadata from a list of runs on NCBI's short read archive. For instance, I'd like to extract the library name ("HS0798") from the following run info:
  https://trace.ncbi.nlm.nih.gov/Traces/sra/sra.cgi?view=run_browser&run=SRR390728
  I have to extract metadata for a few hundred run numbers. Is there a existing command-line tool (sratoolkit, python, perl, bash, etc) that will extract the metadata about the run?
  I could grep it out of a webpage dump, but I notice there are XML schemas provided on the site, which to me indicates that there's some sort of XML API available somewhere. 
  
  user3848: Its got to be the SRA Toolkit
  The way I would do it (a traditional Perl digger) doesn't work on the Tracer NCBI site, at least not readily and dumping the XML is likewise unlikely to succeed.
  
  user3235: I've been told by the support team that there's a way to pull it out in csv form by GETting the following URL, replacing <SRRNUMBER>, with, e.g. SRR000001:
  https://trace.ncbi.nlm.nih.gov/Traces/sra/sra.cgi?save=efetch&db=sra&rettype=runinfo&term=<SRRNUMBER>
  
  So this would be:
  # download the file attachment
  wget -nv -O foo.csv  "https://trace.ncbi.nlm.nih.gov/Traces/sra/sra.cgi?save=efetch&db=sra&rettype=runinfo&term=SRR000001"
  
  # or just get the output on stdout
  curl -v "https://trace.ncbi.nlm.nih.gov/Traces/sra/sra.cgi?save=efetch&db=sra&rettype=runinfo&term=SRR000001"
  
  As an example, the output looks like this for SRR000001. It namely contains additional info on other runs from the same SRX:
  Run,ReleaseDate,LoadDate,spots,bases,spots_with_mates,avgLength,size_MB,AssemblyName,download_path,Experiment,LibraryName,LibraryStrategy,LibrarySelection,LibrarySource,LibraryLayout,InsertSize,InsertDev,Platform,Model,SRAStudy,BioProject,Study_Pubmed_id,ProjectID,Sample,BioSample,SampleType,TaxID,ScientificName,SampleName,g1k_pop_code,source,g1k_analysis_group,Subject_ID,Sex,Disease,Tumor,Affection_Status,Analyte_Type,Histological_Type,Body_Site,CenterName,Submission,dbgap_study_accession,Consent,RunHash,ReadHash
  SRR000001,2008-04-04 14:36:13,2015-04-07 21:54:15,470985,129515135,236041,274,298,,https://sra-download.ncbi.nlm.nih.gov/traces/sra27/SRR/000000/SRR000001,SRX000007,SID2748,WGS,RANDOM,GENOMIC,PAIRED,3000,0,LS454,454 GS FLX,SRP000001,PRJNA33627,1,33627,SRS000100,SAMN00001583,simple,9606,Homo sapiens,NA18505,,,,,female,,no,,,,,454MSC,SRA000197,,public,25D46C95842BCD2F235DDD48070341C4,5FC1B56C9A5332BD21A6FCC6713C3159
  SRR000013,2008-04-04 15:15:44,2012-01-19 15:14:06,478304,131410350,316908,274,307,,https://sra-download.ncbi.nlm.nih.gov/traces/sra4/SRR/000000/SRR000013,SRX000007,SID2748,WGS,RANDOM,GENOMIC,PAIRED,3000,0,LS454,454 GS FLX,SRP000001,PRJNA33627,1,33627,SRS000100,SAMN00001583,simple,9606,Homo sapiens,NA18505,,,,,female,,no,,,,,454MSC,SRA000197,,public,A576F73351E329617F91C92845ADE3BE,0EFE27254FB8790C80E4A8A3A3CCBF1D
  SRR000019,2008-04-04 15:20:15,2012-01-19 15:14:06,537808,147928455,358612,275,347,,https://sra-download.ncbi.nlm.nih.gov/traces/sra4/SRR/000000/SRR000019,SRX000007,SID2748,WGS,RANDOM,GENOMIC,PAIRED,3000,0,LS454,454 GS FLX,SRP000001,PRJNA33627,1,33627,SRS000100,SAMN00001583,simple,9606,Homo sapiens,NA18505,,,,,female,,no,,,,,454MSC,SRA000197,,public,5E42E66B2972A28C3C1B714499B3CE1E,A44F729DF562E11BDA4F950F171085B3
  SRR000033,2008-04-04 15:26:19,2012-01-19 15:14:16,476737,131230950,316780,275,307,,https://sra-download.ncbi.nlm.nih.gov/traces/sra4/SRR/000000/SRR000033,SRX000007,SID2748,WGS,RANDOM,GENOMIC,PAIRED,3000,0,LS454,454 GS FLX,SRP000001,PRJNA33627,1,33627,SRS000100,SAMN00001583,simple,9606,Homo sapiens,NA18505,,,,,female,,no,,,,,454MSC,SRA000197,,public,8D775F0FB1E78F2818166B4E6C7BC320,6C794108B0172AE98323C9A809BE892E
  SRR000047,2008-04-04 15:34:57,2012-01-19 15:14:25,322279,88301883,214744,273,208,,https://sra-download.ncbi.nlm.nih.gov/traces/sra4/SRR/000000/SRR000047,SRX000007,SID2748,WGS,RANDOM,GENOMIC,PAIRED,3000,0,LS454,454 GS FLX,SRP000001,PRJNA33627,1,33627,SRS000100,SAMN00001583,simple,9606,Homo sapiens,NA18505,,,,,female,,no,,,,,454MSC,SRA000197,,public,B4D7BC3DAD22AE58EADC18B2F05B2732,154C9532706F955A63CF685C4D71D9E6
  SRR000049,2008-04-04 15:35:58,2012-01-19 15:14:34,506408,139029601,328595,274,336,,https://sra-download.ncbi.nlm.nih.gov/traces/sra4/SRR/000000/SRR000049,SRX000007,SID2748,WGS,RANDOM,GENOMIC,PAIRED,3000,0,LS454,454 GS FLX,SRP000001,PRJNA33627,1,33627,SRS000100,SAMN00001583,simple,9606,Homo sapiens,NA18505,,,,,female,,no,,,,,454MSC,SRA000197,,public,60D02918CD7686DA084607BFD9D8E6A5,E9A732F2A8F36B8EB1CD38CA41BA8FB5
  SRR000061,2008-04-04 15:41:48,2012-01-19 15:14:37,532884,146330433,351541,274,348,,https://sra-download.ncbi.nlm.nih.gov/traces/sra4/SRR/000000/SRR000061,SRX000007,SID2748,WGS,RANDOM,GENOMIC,PAIRED,3000,0,LS454,454 GS FLX,SRP000001,PRJNA33627,1,33627,SRS000100,SAMN00001583,simple,9606,Homo sapiens,NA18505,,,,,female,,no,,,,,454MSC,SRA000197,,public,8F6E1CA597BCAA19A852036775FA1FAB,2C035CCBD62808A01CEF6EB13BAEB40F
  
  Providing an invalid SRR number will generate an empty HTTP 200 response.
  
  user3235: Here's a very crude script version, which should work without downloading fancy software tools. It will only work as long as the website stays in its current layout.
  It extracts the particular <td> from the webpage, based on a nearby table link ID. and prints it out. 
  Note: This is brittle. It will only work for library names that fit in a single-line of the HTML, and will ignore some webserver errors. The content is also not html-decoded (so its output might show things like &amp; if the field contained html-special characters).
  #!/bin/bash
  
  set -e -o pipefail
  
  # pass the run as a single argument to the script
  run="$1"
  
  # extract library name field. second cell in visual order.
  curl -o - "https://trace.ncbi.nlm.nih.gov/Traces/sra/sra.cgi?view=run_browser&run=$run" 2>/dev/null | \
     grep 'a id="exp-accession"' -A 7 | \
     tail -n +3 | head -n 1 | \
     sed -e 's@^<td>@@' -e 's@</td>$@@'
  
  
  user3577: Entrez Direct is perfect for this sort of thing. You can do something like this:
  $ esearch -db sra -query 'SRR390728' \
    | efetch -format runinfo \
    | cut -f1,12 -d,
  Run,LibraryName
  SRR292241,HS0798
  SRR390728,HS0798
  
  The runinfo table format has 47 fields including experiment accession, project accessions, bioproject and biosample identifiers and such. You may want to explore that a little bit more if you are interested in downloading metadata in addition to the LibraryName.
  Also, if you prefer to parse XML output, you can append -mode xml to the efetch command to retrieve the runinfo table in XML format. xtract tool that comes with the Entrez Direct package can then be used to convert that into a table.
  
original_text: 3235: <p>I wish to extract metadata from a list of runs on NCBI's short read archive. For instance, I'd like to extract the library name ("HS0798") from the following run info:</p>
  
  <p><a href="https://trace.ncbi.nlm.nih.gov/Traces/sra/sra.cgi?view=run_browser&amp;run=SRR390728" rel="noreferrer">https://trace.ncbi.nlm.nih.gov/Traces/sra/sra.cgi?view=run_browser&amp;run=SRR390728</a></p>
  
  <p>I have to extract metadata for a few hundred run numbers. Is there a existing command-line tool (sratoolkit, python, perl, bash, etc) that will extract the <em>metadata</em> about the run?</p>
  
  <p>I could grep it out of a webpage dump, but I notice there are XML schemas provided on the site, which to me indicates that there's some sort of XML API available somewhere. </p>
   3848: <p>Its got to be the <a href="https://trace.ncbi.nlm.nih.gov/Traces/sra/sra.cgi?view=software" rel="nofollow noreferrer">SRA Toolkit</a>
  The way I would do it (a traditional Perl digger) doesn't work on the Tracer NCBI site, at least not readily and dumping the XML is likewise unlikely to succeed.</p>
   3235: <p>I've been told by the support team that there's a way to pull it out in csv form by GETting the following URL, replacing <code>&lt;SRRNUMBER&gt;</code>, with, e.g. <code>SRR000001</code>:</p>
  
  <pre><code>https://trace.ncbi.nlm.nih.gov/Traces/sra/sra.cgi?save=efetch&amp;db=sra&amp;rettype=runinfo&amp;term=&lt;SRRNUMBER&gt;
  </code></pre>
  
  <p>So this would be:</p>
  
  <pre><code># download the file attachment
  wget -nv -O foo.csv  "https://trace.ncbi.nlm.nih.gov/Traces/sra/sra.cgi?save=efetch&amp;db=sra&amp;rettype=runinfo&amp;term=SRR000001"
  
  # or just get the output on stdout
  curl -v "https://trace.ncbi.nlm.nih.gov/Traces/sra/sra.cgi?save=efetch&amp;db=sra&amp;rettype=runinfo&amp;term=SRR000001"
  </code></pre>
  
  <p>As an example, the output looks like this for SRR000001. It namely contains additional info on other runs from the same SRX:</p>
  
  <pre><code>Run,ReleaseDate,LoadDate,spots,bases,spots_with_mates,avgLength,size_MB,AssemblyName,download_path,Experiment,LibraryName,LibraryStrategy,LibrarySelection,LibrarySource,LibraryLayout,InsertSize,InsertDev,Platform,Model,SRAStudy,BioProject,Study_Pubmed_id,ProjectID,Sample,BioSample,SampleType,TaxID,ScientificName,SampleName,g1k_pop_code,source,g1k_analysis_group,Subject_ID,Sex,Disease,Tumor,Affection_Status,Analyte_Type,Histological_Type,Body_Site,CenterName,Submission,dbgap_study_accession,Consent,RunHash,ReadHash
  SRR000001,2008-04-04 14:36:13,2015-04-07 21:54:15,470985,129515135,236041,274,298,,https://sra-download.ncbi.nlm.nih.gov/traces/sra27/SRR/000000/SRR000001,SRX000007,SID2748,WGS,RANDOM,GENOMIC,PAIRED,3000,0,LS454,454 GS FLX,SRP000001,PRJNA33627,1,33627,SRS000100,SAMN00001583,simple,9606,Homo sapiens,NA18505,,,,,female,,no,,,,,454MSC,SRA000197,,public,25D46C95842BCD2F235DDD48070341C4,5FC1B56C9A5332BD21A6FCC6713C3159
  SRR000013,2008-04-04 15:15:44,2012-01-19 15:14:06,478304,131410350,316908,274,307,,https://sra-download.ncbi.nlm.nih.gov/traces/sra4/SRR/000000/SRR000013,SRX000007,SID2748,WGS,RANDOM,GENOMIC,PAIRED,3000,0,LS454,454 GS FLX,SRP000001,PRJNA33627,1,33627,SRS000100,SAMN00001583,simple,9606,Homo sapiens,NA18505,,,,,female,,no,,,,,454MSC,SRA000197,,public,A576F73351E329617F91C92845ADE3BE,0EFE27254FB8790C80E4A8A3A3CCBF1D
  SRR000019,2008-04-04 15:20:15,2012-01-19 15:14:06,537808,147928455,358612,275,347,,https://sra-download.ncbi.nlm.nih.gov/traces/sra4/SRR/000000/SRR000019,SRX000007,SID2748,WGS,RANDOM,GENOMIC,PAIRED,3000,0,LS454,454 GS FLX,SRP000001,PRJNA33627,1,33627,SRS000100,SAMN00001583,simple,9606,Homo sapiens,NA18505,,,,,female,,no,,,,,454MSC,SRA000197,,public,5E42E66B2972A28C3C1B714499B3CE1E,A44F729DF562E11BDA4F950F171085B3
  SRR000033,2008-04-04 15:26:19,2012-01-19 15:14:16,476737,131230950,316780,275,307,,https://sra-download.ncbi.nlm.nih.gov/traces/sra4/SRR/000000/SRR000033,SRX000007,SID2748,WGS,RANDOM,GENOMIC,PAIRED,3000,0,LS454,454 GS FLX,SRP000001,PRJNA33627,1,33627,SRS000100,SAMN00001583,simple,9606,Homo sapiens,NA18505,,,,,female,,no,,,,,454MSC,SRA000197,,public,8D775F0FB1E78F2818166B4E6C7BC320,6C794108B0172AE98323C9A809BE892E
  SRR000047,2008-04-04 15:34:57,2012-01-19 15:14:25,322279,88301883,214744,273,208,,https://sra-download.ncbi.nlm.nih.gov/traces/sra4/SRR/000000/SRR000047,SRX000007,SID2748,WGS,RANDOM,GENOMIC,PAIRED,3000,0,LS454,454 GS FLX,SRP000001,PRJNA33627,1,33627,SRS000100,SAMN00001583,simple,9606,Homo sapiens,NA18505,,,,,female,,no,,,,,454MSC,SRA000197,,public,B4D7BC3DAD22AE58EADC18B2F05B2732,154C9532706F955A63CF685C4D71D9E6
  SRR000049,2008-04-04 15:35:58,2012-01-19 15:14:34,506408,139029601,328595,274,336,,https://sra-download.ncbi.nlm.nih.gov/traces/sra4/SRR/000000/SRR000049,SRX000007,SID2748,WGS,RANDOM,GENOMIC,PAIRED,3000,0,LS454,454 GS FLX,SRP000001,PRJNA33627,1,33627,SRS000100,SAMN00001583,simple,9606,Homo sapiens,NA18505,,,,,female,,no,,,,,454MSC,SRA000197,,public,60D02918CD7686DA084607BFD9D8E6A5,E9A732F2A8F36B8EB1CD38CA41BA8FB5
  SRR000061,2008-04-04 15:41:48,2012-01-19 15:14:37,532884,146330433,351541,274,348,,https://sra-download.ncbi.nlm.nih.gov/traces/sra4/SRR/000000/SRR000061,SRX000007,SID2748,WGS,RANDOM,GENOMIC,PAIRED,3000,0,LS454,454 GS FLX,SRP000001,PRJNA33627,1,33627,SRS000100,SAMN00001583,simple,9606,Homo sapiens,NA18505,,,,,female,,no,,,,,454MSC,SRA000197,,public,8F6E1CA597BCAA19A852036775FA1FAB,2C035CCBD62808A01CEF6EB13BAEB40F
  </code></pre>
  
  <p>Providing an invalid SRR number will generate an empty HTTP 200 response.</p>
   3235: <p>Here's a very crude script version, which should work without downloading fancy software tools. It will only work as long as the website stays in its current layout.</p>
  
  <p>It extracts the particular <code>&lt;td&gt;</code> from the webpage, based on a nearby table link ID. and prints it out. </p>
  
  <p>Note: This is brittle. It will only work for library names that fit in a single-line of the HTML, and will ignore some webserver errors. The content is also not html-decoded (so its output might show things like <code>&amp;amp;</code> if the field contained html-special characters).</p>
  
  <pre><code>#!/bin/bash
  
  set -e -o pipefail
  
  # pass the run as a single argument to the script
  run="$1"
  
  # extract library name field. second cell in visual order.
  curl -o - "https://trace.ncbi.nlm.nih.gov/Traces/sra/sra.cgi?view=run_browser&amp;run=<span class="math-container">$run" 2&gt;/dev/null | \
     grep 'a id="exp-accession"' -A 7 | \
     tail -n +3 | head -n 1 | \
     sed -e 's@^&lt;td&gt;@@' -e 's@&lt;/td&gt;$</span>@@'
  </code></pre>
   3577: <p><a href="https://www.ncbi.nlm.nih.gov/books/NBK179288/" rel="nofollow noreferrer">Entrez Direct</a> is perfect for this sort of thing. You can do something like this:</p>
  <pre><code>$ esearch -db sra -query 'SRR390728' \
    | efetch -format runinfo \
    | cut -f1,12 -d,
  Run,LibraryName
  SRR292241,HS0798
  SRR390728,HS0798
  </code></pre>
  <p>The <code>runinfo</code> table format has 47 fields including experiment accession, project accessions, bioproject and biosample identifiers and such. You may want to explore that a little bit more if you are interested in downloading metadata in addition to the LibraryName.</p>
  <p>Also, if you prefer to parse XML output, you can append <code>-mode xml</code> to the <code>efetch</code> command to retrieve the runinfo table in XML format. <code>xtract</code> tool that comes with the Entrez Direct package can then be used to convert that into a table.</p>
  