{"action":"created","issue":{"url":"https://api.github.com/repos/googlegenomics/dockerflow/issues/7","repository_url":"https://api.github.com/repos/googlegenomics/dockerflow","labels_url":"https://api.github.com/repos/googlegenomics/dockerflow/issues/7/labels{/name}","comments_url":"https://api.github.com/repos/googlegenomics/dockerflow/issues/7/comments","events_url":"https://api.github.com/repos/googlegenomics/dockerflow/issues/7/events","html_url":"https://github.com/googlegenomics/dockerflow/issues/7","id":179910661,"number":7,"title":"Scalability questions","user":{"login":"vardaofthevalier","id":6384889,"avatar_url":"https://avatars.githubusercontent.com/u/6384889?v=3","gravatar_id":"","url":"https://api.github.com/users/vardaofthevalier","html_url":"https://github.com/vardaofthevalier","followers_url":"https://api.github.com/users/vardaofthevalier/followers","following_url":"https://api.github.com/users/vardaofthevalier/following{/other_user}","gists_url":"https://api.github.com/users/vardaofthevalier/gists{/gist_id}","starred_url":"https://api.github.com/users/vardaofthevalier/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vardaofthevalier/subscriptions","organizations_url":"https://api.github.com/users/vardaofthevalier/orgs","repos_url":"https://api.github.com/users/vardaofthevalier/repos","events_url":"https://api.github.com/users/vardaofthevalier/events{/privacy}","received_events_url":"https://api.github.com/users/vardaofthevalier/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":6,"created_at":"2016-09-28T22:47:32Z","updated_at":"2016-09-29T00:41:42Z","closed_at":null,"body":"I have two questions relating to scaling pipelines using Dockerflow:\r\n\r\n1) I notice that the Dockerflow task runner process polls for completion status of the GGP job that it monitors.  One issue that I've run into in the past while developing code to monitor the pipeline jobs is that having too many concurrent processes querying the operations (even if they occasionally sleep) can cause frequent QPS errors which leads to stalled reporting of jobs.  Do you foresee this being an issue here as well, or is there other code somewhere else that handles orchestrating the polling threads and/or the QPS errors?\r\n\r\n2) How many concurrent processes can the Dataflow head node for the workflow handle at once?  I'm just wondering what would happen if I wanted to run an embarrassingly parallel  scatter step (such as running FastQC on potentially tens of thousands of input files) would the Dataflow node be able to handle that without any trouble?  Mainly I think this is important when designing the topology of the graph, so any recommendations you have about the best way to split up the work would be useful.  "},"comment":{"url":"https://api.github.com/repos/googlegenomics/dockerflow/issues/comments/250341329","html_url":"https://github.com/googlegenomics/dockerflow/issues/7#issuecomment-250341329","issue_url":"https://api.github.com/repos/googlegenomics/dockerflow/issues/7","id":250341329,"user":{"login":"pgrosu","id":6555937,"avatar_url":"https://avatars.githubusercontent.com/u/6555937?v=3","gravatar_id":"","url":"https://api.github.com/users/pgrosu","html_url":"https://github.com/pgrosu","followers_url":"https://api.github.com/users/pgrosu/followers","following_url":"https://api.github.com/users/pgrosu/following{/other_user}","gists_url":"https://api.github.com/users/pgrosu/gists{/gist_id}","starred_url":"https://api.github.com/users/pgrosu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pgrosu/subscriptions","organizations_url":"https://api.github.com/users/pgrosu/orgs","repos_url":"https://api.github.com/users/pgrosu/repos","events_url":"https://api.github.com/users/pgrosu/events{/privacy}","received_events_url":"https://api.github.com/users/pgrosu/received_events","type":"User","site_admin":false},"created_at":"2016-09-29T00:41:42Z","updated_at":"2016-09-29T00:41:42Z","body":"No problemo, still very useful with many opportunities for possibilities :)"}}
{"actor":{"display_login":"pgrosu"}}