text: user377: Background: We're increasingly needing some way of storing lots of variant data associated with lots of subjects: think clinical trials and hospital patients, looking for disease-causing or relevant genes. A thousand subjects is where we'd start, there's talk of millions on the horizon. With various genomic medicine initiatives, this is likely a wider need.
  The problem: While there's plenty of platforms out there, it's a rapidly evolving field. It's difficult to get a feel for how (and if) they perform and how they line up against each other:
  
  What's scalable and can handle a lot of data? What sort of limits?
  What's robust and not a teetering pile of hacked-together components?
  What has a large community behind it and is actually used widely?
  What makes for easy access and search from another service? (Commandline, REST or software APIs)
  What sort of variants they handle?
  What sort of parameters can be used in searching? 
  
  Solutions I've seen so far:
  
  BigQ: used with i2b2, but its wider use is unclear
  OpenCGA: looks the most developed, but I've heard complaints about the size of data it spits out
  Using BigQuery over a Google Genomics db: doesn't seem to be a general solution
  Gemini: recommended but is it really scalable and accessible from other services?
  SciDb: a commercial general db
  Quince
  LOVD
  Adam
  Whatever platform DIVAS & RVD run on: which may not be freely available
  Several graphical / graph genome solutions: We (and most other people) are probably not dealing with graph genome data at the moment, but is this a possible solution?
  Roll your own: Frequently recommended but I'm sceptical this is a plausible solution for a large dataset.
  
  Anyone with experience give a review or high-level guide to this platform space?
  
  user37: An epic question. Unfortunately, the short answer is: no, there are no widely used solutions.
  For several thousand samples, BCF2, the binary representation of VCF, should work well. I don't see the need of new tools at this scale. For a larger sample size, ExAC people are using spark-based hail. It keeps all per-sample annotations (like GL, GQ and DP) in addition to genotypes. Hail is at least something heavily used in practice, although mostly by a few groups so far.
  A simpler problem is to store genotypes only. This is sufficient to the majority of end users. There are better approaches to store and query genotypes. GQT, developed by the Gemini team, enables fast query of samples. It allows you to quickly pull samples under certain genotype configurations. As I remember, GQT is orders of magnitude faster than google genomics API to do PCA. Another tool is BGT. It produces a much smaller file and provides fast and convenient queries over sites. Its paper talks about ~32k whole-genome samples. I am in the camp who believe specialized binary formats like GQT and BGT are faster than solutions built on top of generic databases. I would encourage you to have a look if you only want to query genotypes.
  Intel's GenomicDB approaches the problem in a different angle. It does not actually keep a "squared" multi-sample VCF internally. It instead keeps per-sample genotypes/annotations and generates merged VCF on the fly (this is my understanding, which could be wrong). I don't have first-hand experience with GenomicDB, but I think something in this line should be the ultimate solution in the era of 1M samples. I know GATK4 is using it at some step.
  As to others in your list, Gemini might not scale that well, I guess. It is partly the reason why they work on GQT. Last time I checked, BigQuery did not query individual genotypes. It only queries over site statistics. Google genomics APIs access individual genotypes, but I doubt it can be performant. Adam is worth trying. I have not tried, though.
  
original_text: 377: <p><strong>Background:</strong> We're increasingly needing some way of storing lots of variant data associated with lots of subjects: think clinical trials and hospital patients, looking for disease-causing or relevant genes. A thousand subjects is where we'd start, there's talk of millions on the horizon. With various genomic medicine initiatives, this is likely a wider need.</p>
  
  <p><strong>The problem:</strong> While there's plenty of platforms out there, it's a rapidly evolving field. It's difficult to get a feel for how (and if) they perform and how they line up against each other:</p>
  
  <ul>
  <li>What's scalable and can handle a lot of data? What sort of limits?</li>
  <li>What's robust and not a teetering pile of hacked-together components?</li>
  <li>What has a large community behind it and is actually used widely?</li>
  <li>What makes for easy access and search from another service? (Commandline, REST or software APIs)</li>
  <li>What sort of variants they handle?</li>
  <li>What sort of parameters can be used in searching? </li>
  </ul>
  
  <p><strong>Solutions I've seen so far:</strong></p>
  
  <ul>
  <li>BigQ: used with i2b2, but its wider use is unclear</li>
  <li>OpenCGA: looks the most developed, but I've heard complaints about the size of data it spits out</li>
  <li>Using BigQuery over a Google Genomics db: doesn't seem to be a general solution</li>
  <li>Gemini: recommended but is it really scalable and accessible from other services?</li>
  <li>SciDb: a commercial general db</li>
  <li>Quince</li>
  <li>LOVD</li>
  <li>Adam</li>
  <li>Whatever platform DIVAS &amp; RVD run on: which may not be freely available</li>
  <li>Several graphical / graph genome solutions: We (and most other people) are probably not dealing with graph genome data at the moment, but is this a possible solution?</li>
  <li>Roll your own: Frequently recommended but I'm sceptical this is a plausible solution for a large dataset.</li>
  </ul>
  
  <p>Anyone with experience give a review or high-level guide to this platform space?</p>
   37: <p>An epic question. Unfortunately, the short answer is: no, there are no widely used solutions.</p>
  
  <p>For several thousand samples, BCF2, the binary representation of VCF, should work well. I don't see the need of new tools at this scale. For a larger sample size, ExAC people are using spark-based hail. It keeps all per-sample annotations (like GL, GQ and DP) in addition to genotypes. Hail is at least something heavily used in practice, although mostly by a few groups so far.</p>
  
  <p>A simpler problem is to store genotypes only. This is sufficient to the majority of end users. There are better approaches to store and query genotypes. GQT, developed by the Gemini team, enables fast query of samples. It allows you to quickly pull samples under certain genotype configurations. As I remember, GQT is orders of magnitude faster than google genomics API to do PCA. Another tool is BGT. It produces a much smaller file and provides fast and convenient queries over sites. Its paper talks about ~32k whole-genome samples. I am in the camp who believe specialized binary formats like GQT and BGT are faster than solutions built on top of generic databases. I would encourage you to have a look if you only want to query genotypes.</p>
  
  <p>Intel's GenomicDB approaches the problem in a different angle. It does not actually keep a "squared" multi-sample VCF internally. It instead keeps per-sample genotypes/annotations and generates merged VCF on the fly (this is my understanding, which could be wrong). I don't have first-hand experience with GenomicDB, but I think something in this line should be the ultimate solution in the era of 1M samples. I know GATK4 is using it at some step.</p>
  
  <p>As to others in your list, Gemini might not scale that well, I guess. It is partly the reason why they work on GQT. Last time I checked, BigQuery did not query individual genotypes. It only queries over site statistics. Google genomics APIs access individual genotypes, but I doubt it can be performant. Adam is worth trying. I have not tried, though.</p>
  