text: user1196: I am bench-marking the performance of ADAM/Avocado on NGS whole-genome sequencing datasets. I am trying to repeat the tests reported in Frank Nofthat's (author of ADAM and Avocado) thesis, however, I am running into an issue acquiring the dbSNP database to use for Base Quality Score Recalibration.
  The Base Quality Score Recalibration command (page 128-129):
  adam-submit -- transformAlignments aln.adam bqsr.adam -recalibrate_base_qualities -known_snps dbsnp.adam
  
  requires dbsnp.adam, which is described as "the dnSNP database" (converted to ADAM format) with reference to the NCBI database of genetic variation. This "database" can be found at this FTP site however I do not know which parts of these files to download and convert to ADAM in order to perform this step of the transformation pipeline. 
  Also, in order to use this genetic variation databaes do I need to make sure that the reference genome of "known variants" is the same genome that the sequencing data was aligned to?
  Thank you.
  
  user383: In the dbSNP FTP site you linked, you need to go into organisms and select your organism of interest (human obviously). You can then select which release of the SNP database you want (eg. 150 vs 151) [the modified dates hint at when they were released] and which genome build you want (eg. hg37 vs hg38).
  Below I am looking at the .vcf files. I assume this what you want because they can be converted to ADAM format via the adam vcf2adam command.
  For example human (newest genome and newest dbSNP release):
  ftp://ftp.ncbi.nlm.nih.gov/snp/organisms/human_9606_b151_GRCh38p7/VCF/
  Your next decision is whether or not you want to use ALL the annotated SNPs in the database or just the common ones. For the former you choose the All_20180418 file and for the later the common_all_20180418 file. I'm pretty sure the convention here is that the date in the filename is when it was uploaded/updated.
  In your case I'm not sure what makes the most sense. Firstly, are you trying to replicate the initial conditions of when that software was published or do you want simply to benchmark the method with the best available data. Besides that, I imagine using all them is computationally much more taxing and if you are just coming up with general trend then using the common ones seems viable.
  
original_text: 1196: <p>I am bench-marking the performance of <a href="https://github.com/bigdatagenomics/adam" rel="nofollow noreferrer">ADAM</a>/<a href="https://github.com/bigdatagenomics/avocado" rel="nofollow noreferrer">Avocado</a> on NGS whole-genome sequencing datasets. I am trying to repeat the tests reported in Frank Nofthat's (author of <a href="https://github.com/bigdatagenomics/adam" rel="nofollow noreferrer">ADAM</a> and <a href="https://github.com/bigdatagenomics/avocado" rel="nofollow noreferrer">Avocado</a>) <a href="https://www2.eecs.berkeley.edu/Pubs/TechRpts/2017/EECS-2017-204.html" rel="nofollow noreferrer">thesis</a>, however, I am running into an issue acquiring the dbSNP database to use for Base Quality Score Recalibration.</p>
  
  <p>The Base Quality Score Recalibration command (page 128-129):</p>
  
  <pre><code>adam-submit -- transformAlignments aln.adam bqsr.adam -recalibrate_base_qualities -known_snps dbsnp.adam
  </code></pre>
  
  <p>requires <code>dbsnp.adam</code>, which is described as "the dnSNP database" (converted to ADAM format) with reference to <a href="https://www.ncbi.nlm.nih.gov/pubmed/11125122" rel="nofollow noreferrer">the NCBI database of genetic variation</a>. This "database" can be found at <a href="ftp://ftp.ncbi.nih.gov/snp/" rel="nofollow noreferrer">this FTP site</a> however I do not know which parts of these files to download and convert to ADAM in order to perform this step of the transformation pipeline. </p>
  
  <p>Also, in order to use this genetic variation databaes do I need to make sure that the reference genome of "known variants" is the same genome that the sequencing data was aligned to?</p>
  
  <p>Thank you.</p>
   383: <p>In the <a href="ftp://ftp.ncbi.nih.gov/snp/" rel="nofollow noreferrer">dbSNP FTP site</a> you linked, you need to go into organisms and select your organism of interest (human obviously). You can then select which release of the SNP database you want (eg. 150 vs 151) [the <em>modified</em> dates hint at when they were released] and which genome build you want (eg. hg37 vs hg38).</p>
  
  <p>Below I am looking at the <em>.vcf</em> files. I assume this what you want because they can be converted to ADAM format via the <code>adam vcf2adam</code> command.</p>
  
  <p>For example human (newest genome and newest dbSNP release):
  <a href="ftp://ftp.ncbi.nlm.nih.gov/snp/organisms/human_9606_b151_GRCh38p7/VCF/" rel="nofollow noreferrer">ftp://ftp.ncbi.nlm.nih.gov/snp/organisms/human_9606_b151_GRCh38p7/VCF/</a></p>
  
  <p>Your next decision is whether or not you want to use ALL the annotated SNPs in the database or just the common ones. For the former you choose the <em>All_20180418</em> file and for the later the <em>common_all_20180418</em> file. I'm pretty sure the convention here is that the date in the filename is when it was uploaded/updated.</p>
  
  <p>In your case I'm not sure what makes the most sense. Firstly, are you trying to replicate the initial conditions of when that software was published or do you want simply to benchmark the method with the best available data. Besides that, I imagine using all them is computationally much more taxing and if you are just coming up with general trend then using the common ones seems viable.</p>
  