{"action":"created","issue":{"url":"https://api.github.com/repos/ssadedin/bpipe/issues/185","repository_url":"https://api.github.com/repos/ssadedin/bpipe","labels_url":"https://api.github.com/repos/ssadedin/bpipe/issues/185/labels{/name}","comments_url":"https://api.github.com/repos/ssadedin/bpipe/issues/185/comments","events_url":"https://api.github.com/repos/ssadedin/bpipe/issues/185/events","html_url":"https://github.com/ssadedin/bpipe/issues/185","id":170656843,"number":185,"title":"SGE, possible \"lost\" executors in highly parallel pipeline","user":{"login":"gdevenyi","id":3001850,"avatar_url":"https://avatars.githubusercontent.com/u/3001850?v=3","gravatar_id":"","url":"https://api.github.com/users/gdevenyi","html_url":"https://github.com/gdevenyi","followers_url":"https://api.github.com/users/gdevenyi/followers","following_url":"https://api.github.com/users/gdevenyi/following{/other_user}","gists_url":"https://api.github.com/users/gdevenyi/gists{/gist_id}","starred_url":"https://api.github.com/users/gdevenyi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gdevenyi/subscriptions","organizations_url":"https://api.github.com/users/gdevenyi/orgs","repos_url":"https://api.github.com/users/gdevenyi/repos","events_url":"https://api.github.com/users/gdevenyi/events{/privacy}","received_events_url":"https://api.github.com/users/gdevenyi/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2016-08-11T14:17:35Z","updated_at":"2016-09-20T21:43:01Z","closed_at":null,"body":"Thanks to the high memory fixes, I can now run a pipeline with 8+ stages on 1000+ inputs without any memory issues (see https://github.com/CobraLab/minc-bpipe-library for pipeline)\r\n\r\nHowever, I seem to be running into some kind of accounting issue with executors which I can't quite place.\r\n\r\nI run a pipeline like this:\r\n\r\n```groovy\r\nbeast = segment {\r\n    beastnormalize + beastmask + beastextract\r\n}\r\n\r\npreprocess = segment {\r\n\tclean_and_center + n4correct + linear_bestlinreg + cutneckapplyautocrop + beast + QC\r\n}\r\n\r\nrun {\r\n \"%.mnc\" * [preprocess]\r\n}\r\n```\r\n\r\nSome of the above stages use more than one \"proc\" during run, which is properly submitted to a SGE smp parallel environment.\r\n\r\nNow running\r\n```sh\r\n# 240 slots available in the cluster\r\n$ bpipe run -n240 /path/to/pipeline.bpipe /path/to/inputs/*.mnc\r\n```\r\n\r\nWhat I have found is, at the end of the \"beast\" segment, the pipeline will sit waiting for stages in the \"beast\" segment to complete before submitting more jobs. Even if there are unfinished prior stages for other inputs and other finished \"beast\" stage outputs, \"QC\" doesn't start.\r\n\r\nThe ends up with some small number of inputs running on the cluster with bpipe waiting for them to finish before submitting other jobs for both prior and subsequent stages.\r\n\r\nThis leads me to believe there's an accounting issue with executors/procs/jobs.\r\n\r\nCan you suggest any tests for this?\r\n"},"comment":{"url":"https://api.github.com/repos/ssadedin/bpipe/issues/comments/248443812","html_url":"https://github.com/ssadedin/bpipe/issues/185#issuecomment-248443812","issue_url":"https://api.github.com/repos/ssadedin/bpipe/issues/185","id":248443812,"user":{"login":"ssadedin","id":138868,"avatar_url":"https://avatars.githubusercontent.com/u/138868?v=3","gravatar_id":"","url":"https://api.github.com/users/ssadedin","html_url":"https://github.com/ssadedin","followers_url":"https://api.github.com/users/ssadedin/followers","following_url":"https://api.github.com/users/ssadedin/following{/other_user}","gists_url":"https://api.github.com/users/ssadedin/gists{/gist_id}","starred_url":"https://api.github.com/users/ssadedin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ssadedin/subscriptions","organizations_url":"https://api.github.com/users/ssadedin/orgs","repos_url":"https://api.github.com/users/ssadedin/repos","events_url":"https://api.github.com/users/ssadedin/events{/privacy}","received_events_url":"https://api.github.com/users/ssadedin/received_events","type":"User","site_admin":false},"created_at":"2016-09-20T21:43:01Z","updated_at":"2016-09-20T21:43:01Z","body":"Change aaa835a is relevant to this issue."}}
{"actor":{"display_login":"ssadedin"}}