{"action":"created","issue":{"url":"https://api.github.com/repos/elixir-no-nels/rbFlow-Germline/issues/22","repository_url":"https://api.github.com/repos/elixir-no-nels/rbFlow-Germline","labels_url":"https://api.github.com/repos/elixir-no-nels/rbFlow-Germline/issues/22/labels{/name}","comments_url":"https://api.github.com/repos/elixir-no-nels/rbFlow-Germline/issues/22/comments","events_url":"https://api.github.com/repos/elixir-no-nels/rbFlow-Germline/issues/22/events","html_url":"https://github.com/elixir-no-nels/rbFlow-Germline/issues/22","id":389737133,"node_id":"MDU6SXNzdWUzODk3MzcxMzM=","number":22,"title":"Spark tools are officially recommended to not be used for production","user":{"login":"oskarvid","id":17961690,"node_id":"MDQ6VXNlcjE3OTYxNjkw","avatar_url":"https://avatars0.githubusercontent.com/u/17961690?v=4","gravatar_id":"","url":"https://api.github.com/users/oskarvid","html_url":"https://github.com/oskarvid","followers_url":"https://api.github.com/users/oskarvid/followers","following_url":"https://api.github.com/users/oskarvid/following{/other_user}","gists_url":"https://api.github.com/users/oskarvid/gists{/gist_id}","starred_url":"https://api.github.com/users/oskarvid/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/oskarvid/subscriptions","organizations_url":"https://api.github.com/users/oskarvid/orgs","repos_url":"https://api.github.com/users/oskarvid/repos","events_url":"https://api.github.com/users/oskarvid/events{/privacy}","received_events_url":"https://api.github.com/users/oskarvid/received_events","type":"User","site_admin":false},"labels":[{"id":949273715,"node_id":"MDU6TGFiZWw5NDkyNzM3MTU=","url":"https://api.github.com/repos/elixir-no-nels/rbFlow-Germline/labels/question","name":"question","color":"d876e3","default":true}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/elixir-no-nels/rbFlow-Germline/milestones/1","html_url":"https://github.com/elixir-no-nels/rbFlow-Germline/milestone/1","labels_url":"https://api.github.com/repos/elixir-no-nels/rbFlow-Germline/milestones/1/labels","id":3586897,"node_id":"MDk6TWlsZXN0b25lMzU4Njg5Nw==","number":1,"title":"First rbFlowLite release","description":"","creator":{"login":"oskarvid","id":17961690,"node_id":"MDQ6VXNlcjE3OTYxNjkw","avatar_url":"https://avatars0.githubusercontent.com/u/17961690?v=4","gravatar_id":"","url":"https://api.github.com/users/oskarvid","html_url":"https://github.com/oskarvid","followers_url":"https://api.github.com/users/oskarvid/followers","following_url":"https://api.github.com/users/oskarvid/following{/other_user}","gists_url":"https://api.github.com/users/oskarvid/gists{/gist_id}","starred_url":"https://api.github.com/users/oskarvid/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/oskarvid/subscriptions","organizations_url":"https://api.github.com/users/oskarvid/orgs","repos_url":"https://api.github.com/users/oskarvid/repos","events_url":"https://api.github.com/users/oskarvid/events{/privacy}","received_events_url":"https://api.github.com/users/oskarvid/received_events","type":"User","site_admin":false},"open_issues":9,"closed_issues":9,"state":"open","created_at":"2018-08-21T12:39:07Z","updated_at":"2018-12-11T12:16:48Z","due_on":null,"closed_at":null},"comments":4,"created_at":"2018-12-11T12:16:47Z","updated_at":"2018-12-13T10:01:22Z","closed_at":null,"author_association":"COLLABORATOR","body":"Technically we cannot guarantee to the end user that the output from the spark enabled tools is of high quality. The log file from HaplotypeCallerSpark says this:  \r\n>The output of this tool DOES NOT match the output of HaplotypeCaller. \r\nIt is under development and should not be used for production work. \r\nFor evaluation only.\r\nUse the non-spark HaplotypeCaller if you care about the results.\r\n\r\nBaseRecalibratorSpark says this:\r\n>!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!\r\n>\r\n>Warning: BaseRecalibratorSpark is a BETA tool and is not yet ready for use in production\r\n>\r\n>!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!\r\n\r\nIf we want to ship with the spark enabled versions while they're still in beta status I think we should inform the end user that the results aren't guaranteed to have the same quality as the non spark HaplotypeCaller version.\r\n"},"comment":{"url":"https://api.github.com/repos/elixir-no-nels/rbFlow-Germline/issues/comments/446910371","html_url":"https://github.com/elixir-no-nels/rbFlow-Germline/issues/22#issuecomment-446910371","issue_url":"https://api.github.com/repos/elixir-no-nels/rbFlow-Germline/issues/22","id":446910371,"node_id":"MDEyOklzc3VlQ29tbWVudDQ0NjkxMDM3MQ==","user":{"login":"oskarvid","id":17961690,"node_id":"MDQ6VXNlcjE3OTYxNjkw","avatar_url":"https://avatars0.githubusercontent.com/u/17961690?v=4","gravatar_id":"","url":"https://api.github.com/users/oskarvid","html_url":"https://github.com/oskarvid","followers_url":"https://api.github.com/users/oskarvid/followers","following_url":"https://api.github.com/users/oskarvid/following{/other_user}","gists_url":"https://api.github.com/users/oskarvid/gists{/gist_id}","starred_url":"https://api.github.com/users/oskarvid/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/oskarvid/subscriptions","organizations_url":"https://api.github.com/users/oskarvid/orgs","repos_url":"https://api.github.com/users/oskarvid/repos","events_url":"https://api.github.com/users/oskarvid/events{/privacy}","received_events_url":"https://api.github.com/users/oskarvid/received_events","type":"User","site_admin":false},"created_at":"2018-12-13T10:01:22Z","updated_at":"2018-12-13T10:01:22Z","author_association":"COLLABORATOR","body":"It's possible to fork processes in ruby: http://rubyforadmins.com/background-jobs  \r\nSo I think we can make a for loop, spawn one process per sub-interval list, wait for them to finish and then proceed with the next tool. We would need to implement this workflow in rbFlow: https://raw.githubusercontent.com/oskarvid/snakemake_germline/master/dag.png\r\n\r\nIn particular we need to do the following changes:  \r\n1) Add a FastqToSam step to create an unmapped bam file\r\n2) Combine the mapped and unmapped bam files with MergeBamAlignment\r\n3) Combine the MergeBamAlignment outputs with MarkDuplicates\r\n4) Scatter BaseRecalibration by running a for loop that forks each process to the background\r\n5) Gather the outputs with GatherBQSR\r\n6) Repeat the same principle in step 4 and 5 for ApplyBQSR and HaplotypeCaller, we can do this for GenotypeGVCFs too if we want to, it saves about 30 minutes I think.\r\n\r\nI understand that this is a lot of work, but it's probably possible, so I want to at least present the possibility so that we know it's a potential option. "}}
{"actor":{"display_login":"oskarvid"}}