{"action":"created","issue":{"url":"https://api.github.com/repos/chapmanb/bcbio-nextgen/issues/799","repository_url":"https://api.github.com/repos/chapmanb/bcbio-nextgen","labels_url":"https://api.github.com/repos/chapmanb/bcbio-nextgen/issues/799/labels{/name}","comments_url":"https://api.github.com/repos/chapmanb/bcbio-nextgen/issues/799/comments","events_url":"https://api.github.com/repos/chapmanb/bcbio-nextgen/issues/799/events","html_url":"https://github.com/chapmanb/bcbio-nextgen/issues/799","id":63706581,"number":799,"title":"bcbio order of task processing seems to be wasteful concerning storage requirements","user":{"login":"schelhorn","id":5024959,"avatar_url":"https://avatars.githubusercontent.com/u/5024959?v=3","gravatar_id":"","url":"https://api.github.com/users/schelhorn","html_url":"https://github.com/schelhorn","followers_url":"https://api.github.com/users/schelhorn/followers","following_url":"https://api.github.com/users/schelhorn/following{/other_user}","gists_url":"https://api.github.com/users/schelhorn/gists{/gist_id}","starred_url":"https://api.github.com/users/schelhorn/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/schelhorn/subscriptions","organizations_url":"https://api.github.com/users/schelhorn/orgs","repos_url":"https://api.github.com/users/schelhorn/repos","events_url":"https://api.github.com/users/schelhorn/events{/privacy}","received_events_url":"https://api.github.com/users/schelhorn/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2015-03-23T12:04:45Z","updated_at":"2016-12-16T22:39:56Z","closed_at":"2016-12-16T22:39:55Z","body":"Dear Brad & other contributors,\n\nthe following is more a request for clarification or a feature request than a bug; however it may be of some interest in others and may also result in code changes, that's why I decided to list it here.\n\nIt seems to be as if the order of processing of bcbio currently does not try to limit storage requirements since the same task (e.g., alignment) seems to be scheduled in parallel for all samples rather than only for a subset of samples that can be processed until completion. In other words, if there is a order of tasks n in 1:N for each sample that have to be completed in serial order for completing the analysis of that sample, bcbio seems to schedule task n for completed for _all_ samples before before task n+1 is started, again for all samples. As a result, the maximal storage requirements for a bcbio analysis seem to be quite high, since first all alignments and their post-processed versions have to be stored in the working directory before the result files, for all samples in parallel, are generated. \n\nIn contrast, I suggest it would be more parsimonious to run as many samples until completion as can be processed in parallel (CPU-wise and memory-wise), then upload the results to the final `upload` destination and finally remove all data pertaining to these samples from the working directory (perhaps by leaving behind a lock-file indicating a sample has finished, to allow for a clean restart after errors), before  scheduling tasks for the next set of samples. In this manner, storage requirements would be significantly lessened and scalability improved. I realize that for some analysis steps (e.g., joint variant calling, but there may be others) certain output data have to be preserved well after completion of a sample to allow for across-sample analyses; however, these data could then be retained in the working directory as required and only deleted after they are not longer required.\n\nThe reason why I would prefer such a strategy is that high-bandwidth clustered network storage is expensive and at our organization we do not have enough of it for moving a couple of hundred of exomes across it while also storing all alignments in parallel. Many labs have a tiered storage architecture of (and your notation may differ here) tier 1 node-local storage (e.g., raid-0 SSDs), tier 2 clustered network storage (e.g., lustre, or in principle also HDFS), tier 3 non-clustered network storage (e.g., NFS), and tier 4 archive storage (more slow NFS, S3, tape). However, for highly parallel processing on a job scheduler you are basically limited to using tier 2 storage (plus tier 1 for local temporary files) for performance and parallel access reasons.\n\nPlease correct me of bcbio already follows such a execution strategy, or offers an option for doing so – my observations are relying on limited experience with a bcbio test instance we plan to move to production eventually, and I may have misinterpreted the data. Either way, an option for providing automatic deletion of intermediary files in the working directory that are not used by other analysis steps and also not uploaded to the final `upload` destination would be valuable to save storage, even if the execution strategy is fine or too much hassle to change.\n"},"comment":{"url":"https://api.github.com/repos/chapmanb/bcbio-nextgen/issues/comments/267713497","html_url":"https://github.com/chapmanb/bcbio-nextgen/issues/799#issuecomment-267713497","issue_url":"https://api.github.com/repos/chapmanb/bcbio-nextgen/issues/799","id":267713497,"user":{"login":"lpantano","id":1621788,"avatar_url":"https://avatars.githubusercontent.com/u/1621788?v=3","gravatar_id":"","url":"https://api.github.com/users/lpantano","html_url":"https://github.com/lpantano","followers_url":"https://api.github.com/users/lpantano/followers","following_url":"https://api.github.com/users/lpantano/following{/other_user}","gists_url":"https://api.github.com/users/lpantano/gists{/gist_id}","starred_url":"https://api.github.com/users/lpantano/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lpantano/subscriptions","organizations_url":"https://api.github.com/users/lpantano/orgs","repos_url":"https://api.github.com/users/lpantano/repos","events_url":"https://api.github.com/users/lpantano/events{/privacy}","received_events_url":"https://api.github.com/users/lpantano/received_events","type":"User","site_admin":false},"created_at":"2016-12-16T22:39:55Z","updated_at":"2016-12-16T22:39:55Z","body":"Hi \r\n\r\nI am closing this because it seems an old issue. Come back with other suggestion since the pipeline changed a lot since then.\r\n\r\ncheers"}}
{"actor":{"display_login":"lpantano"}}