{"action":"created","issue":{"url":"https://api.github.com/repos/chapmanb/bcbio-nextgen/issues/1556","repository_url":"https://api.github.com/repos/chapmanb/bcbio-nextgen","labels_url":"https://api.github.com/repos/chapmanb/bcbio-nextgen/issues/1556/labels{/name}","comments_url":"https://api.github.com/repos/chapmanb/bcbio-nextgen/issues/1556/comments","events_url":"https://api.github.com/repos/chapmanb/bcbio-nextgen/issues/1556/events","html_url":"https://github.com/chapmanb/bcbio-nextgen/issues/1556","id":176667608,"number":1556,"title":"Would it be possible to run the QC stage in parallel?","user":{"login":"NeillGibson","id":15608993,"avatar_url":"https://avatars.githubusercontent.com/u/15608993?v=3","gravatar_id":"","url":"https://api.github.com/users/NeillGibson","html_url":"https://github.com/NeillGibson","followers_url":"https://api.github.com/users/NeillGibson/followers","following_url":"https://api.github.com/users/NeillGibson/following{/other_user}","gists_url":"https://api.github.com/users/NeillGibson/gists{/gist_id}","starred_url":"https://api.github.com/users/NeillGibson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/NeillGibson/subscriptions","organizations_url":"https://api.github.com/users/NeillGibson/orgs","repos_url":"https://api.github.com/users/NeillGibson/repos","events_url":"https://api.github.com/users/NeillGibson/events{/privacy}","received_events_url":"https://api.github.com/users/NeillGibson/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":14,"created_at":"2016-09-13T15:10:10Z","updated_at":"2016-09-23T14:39:57Z","closed_at":null,"body":"Hi,\r\n\r\nWould it be possible to run the QC stage in parallel? \r\n\r\nI like the new (for me) MultiQC report very much but it takes a bit long to create all the QC files compared to the rest of the analysis stages. \r\n\r\nI think the QC stage currently runs sequential for all samples. While all other analysis stages run parallel per sample or per genome region. \r\n\r\n Is it intended for the QC stage to run sequentially per sample?  This older issue https://github.com/chapmanb/bcbio-nextgen/issues/1175 mentions that at that time it was difficult to parallelize stages after variants calling is finished. \r\n\r\nFor a recent data set of 42 samples that I processed with bcbio 0.9.9  (bwa+ freebayes) running the QC stage takes the longest of all stages. \r\n\r\n\r\nAnalysis stage  | Total wall time (hh:mm) | Total wall time / nr samples (hh:mm)\r\n------------- | -------------|--------------------|\r\nprep_align_inputs | 03:33| 00:05 |\r\nprocess_alignment  | 26:00|00:37|\r\ncallable regions|14:00|00:20|\r\nvariant calling|25:00|00:35|\r\npostprocess_variants|03:00|00:04\r\nQC|  42:00  | 01:00\r\n\r\nRunning the QC stage fully in parallel would reduce the total analysis time from 113 hours to 72 hours. A reduction of ca 36% in total wall time.  This percentage would become even much more significant for sets of samples in the hundreds. \r\n\r\nThe QC stage splits to the following sub tasks run sequentially per sample. \r\n\r\nQC Analysis task| Total wall time (hh:mm) \r\n------------- | -------------|\r\nDownsample bam  | 00:02| \r\nFastQC  | 00:01||\r\nSamtools stats|00:11|\r\nsample VCF stats |00:41|\r\n\r\nAnother option would be to  convert the VCF.gz file to a temporary(?) BCF.gz file using ```bcftools view -O b ``` . \r\n\r\nThis would take around 2 hours for my 30 GB vcf.gz    (running the conversion now)\r\nBut this would be more than compensated by reducing the time needed for the ```sample VCF stats``` by 50% or 75% ( will test tomorrow. )\r\n\r\nConversion to BCF.gz is something we already would like  do for every bcbio output to speed up other bcftools queries that we do downstream significantly. We do still need the vcf.gz though because more tools support it (  I am guessing this is the reason for bcbio outputting vcf.gz )\r\n\r\nI do like the VCF stats and the MultiQC report so please don't remove the VCF stats  :) \r\n\r\nThank you. "},"comment":{"url":"https://api.github.com/repos/chapmanb/bcbio-nextgen/issues/comments/249210803","html_url":"https://github.com/chapmanb/bcbio-nextgen/issues/1556#issuecomment-249210803","issue_url":"https://api.github.com/repos/chapmanb/bcbio-nextgen/issues/1556","id":249210803,"user":{"login":"NeillGibson","id":15608993,"avatar_url":"https://avatars.githubusercontent.com/u/15608993?v=3","gravatar_id":"","url":"https://api.github.com/users/NeillGibson","html_url":"https://github.com/NeillGibson","followers_url":"https://api.github.com/users/NeillGibson/followers","following_url":"https://api.github.com/users/NeillGibson/following{/other_user}","gists_url":"https://api.github.com/users/NeillGibson/gists{/gist_id}","starred_url":"https://api.github.com/users/NeillGibson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/NeillGibson/subscriptions","organizations_url":"https://api.github.com/users/NeillGibson/orgs","repos_url":"https://api.github.com/users/NeillGibson/repos","events_url":"https://api.github.com/users/NeillGibson/events{/privacy}","received_events_url":"https://api.github.com/users/NeillGibson/received_events","type":"User","site_admin":false},"created_at":"2016-09-23T14:39:57Z","updated_at":"2016-09-23T14:39:57Z","body":"Thank you for fixing this parallelization issue.  \r\nI will try to test it end of next but week but looks like it should do the job. "}}
{"actor":{"display_login":"NeillGibson"}}