{"action":"closed","issue":{"url":"https://api.github.com/repos/bigdatagenomics/adam/issues/1179","repository_url":"https://api.github.com/repos/bigdatagenomics/adam","labels_url":"https://api.github.com/repos/bigdatagenomics/adam/issues/1179/labels{/name}","comments_url":"https://api.github.com/repos/bigdatagenomics/adam/issues/1179/comments","events_url":"https://api.github.com/repos/bigdatagenomics/adam/issues/1179/events","html_url":"https://github.com/bigdatagenomics/adam/issues/1179","id":178118193,"number":1179,"title":"How to read glob of multiple parquet Genotype","user":{"login":"jpdna","id":7423296,"avatar_url":"https://avatars.githubusercontent.com/u/7423296?v=3","gravatar_id":"","url":"https://api.github.com/users/jpdna","html_url":"https://github.com/jpdna","followers_url":"https://api.github.com/users/jpdna/followers","following_url":"https://api.github.com/users/jpdna/following{/other_user}","gists_url":"https://api.github.com/users/jpdna/gists{/gist_id}","starred_url":"https://api.github.com/users/jpdna/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jpdna/subscriptions","organizations_url":"https://api.github.com/users/jpdna/orgs","repos_url":"https://api.github.com/users/jpdna/repos","events_url":"https://api.github.com/users/jpdna/events{/privacy}","received_events_url":"https://api.github.com/users/jpdna/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2016-09-20T17:03:27Z","updated_at":"2016-09-21T15:00:12Z","closed_at":"2016-09-21T15:00:12Z","body":"Is it possible to load the data of several parquet files storing bdg-formats Genotype into a single GenotypeRDD?\r\nI want something equivalent to: sc.loadVcf(\"*.vcf\") typeglob but for ADAM Parquet files.\r\n\r\nSee discussions below from email:\r\n\r\nOn Tue, Sep 20, 2016 at 11:19 AM, Justin Paschall <justinpaschalldna@gmail.com> wrote:\r\nFor set of single samples VCFs Michael pointed out using  sc.loadVcf(\"*.vcf\") can read in a bunch of single samples VCFs into a single RDD[VariantContext] of a VariantContextRDD\r\n\r\nSimilarly - is there any sense to to converting single-sample VCFs to single sample Parquet files of [Genotype]  as a data source of reference?\r\n\r\nHowever, I am not sure there is a way to typeglob load a set of many individual ADAM parquet genotypes files upon loading, in the same way that we can load a bunch of VCF with the typeglob?\r\n\r\nIt is tricky to get the globs in correctly without having them expanded by the shell; it appears that even my tricks don't work for .adam files\r\n\r\n$ ./bin/adam-submit transform \"*.adam\" -single combined.sam\r\nCommand body threw exception:\r\njava.io.FileNotFoundException: Couldn't find any files matching *.adam\r\nException in thread \"main\" java.io.FileNotFoundException: Couldn't find any files matching *.adam\r\n    at org.bdgenomics.adam.rdd.ADAMContext.getFsAndFilesWithFilter(ADAMContext.scala:391)\r\n    at org.bdgenomics.adam.rdd.ADAMContext.loadAvroSequences(ADAMContext.scala:207)\r\n    at org.bdgenomics.adam.rdd.ADAMContext.loadParquetAlignments(ADAMContext.scala:637)\r\n\r\n$ ./bin/adam-submit transform \"file:///`pwd`/*.adam\" -single combined.sam\r\nCommand body threw exception:\r\njava.io.FileNotFoundException: Couldn't find any files matching file:///Users/heuermh/working/adam/*.adam\r\nException in thread \"main\" java.io.FileNotFoundException: Couldn't find any files matching file:///Users/heuermh/working/adam/*.adam\r\n    at org.bdgenomics.adam.rdd.ADAMContext.getFsAndFilesWithFilter(ADAMContext.scala:391)\r\n    at org.bdgenomics.adam.rdd.ADAMContext.loadAvroSequences(ADAMContext.scala:207)\r\n    at org.bdgenomics.adam.rdd.ADAMContext.loadParquetAlignments(ADAMContext.scala:637)\r\n\r\nWill have to dig in the new code changes to look (getFsAndFilesWithFilter).\r\n\r\n \r\nThe VCF type glob works, but it is slower than I hoped, 10 minutes to load/count on 100 samples of chr22 on my workstation (testing on cluster shortly), and I want to make sure that I am not missing out on a prior one-time conversion to parquet on a per-sample-file basis if that would be better.\r\n\r\n\r\n"}}
{"actor":{"display_login":"heuermh"}}