{"action":"closed","issue":{"url":"https://api.github.com/repos/ssadedin/bpipe/issues/120","repository_url":"https://api.github.com/repos/ssadedin/bpipe","labels_url":"https://api.github.com/repos/ssadedin/bpipe/issues/120/labels{/name}","comments_url":"https://api.github.com/repos/ssadedin/bpipe/issues/120/comments","events_url":"https://api.github.com/repos/ssadedin/bpipe/issues/120/events","html_url":"https://github.com/ssadedin/bpipe/issues/120","id":52377024,"number":120,"title":"Bpipe process has extremely high memory usage for multi-stage highly parallel pipelines (especially during resume)","user":{"login":"gdevenyi","id":3001850,"avatar_url":"https://avatars.githubusercontent.com/u/3001850?v=3","gravatar_id":"","url":"https://api.github.com/users/gdevenyi","html_url":"https://github.com/gdevenyi","followers_url":"https://api.github.com/users/gdevenyi/followers","following_url":"https://api.github.com/users/gdevenyi/following{/other_user}","gists_url":"https://api.github.com/users/gdevenyi/gists{/gist_id}","starred_url":"https://api.github.com/users/gdevenyi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gdevenyi/subscriptions","organizations_url":"https://api.github.com/users/gdevenyi/orgs","repos_url":"https://api.github.com/users/gdevenyi/repos","events_url":"https://api.github.com/users/gdevenyi/events{/privacy}","received_events_url":"https://api.github.com/users/gdevenyi/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/ssadedin/bpipe/labels/enhancement","name":"enhancement","color":"84b6eb"},{"url":"https://api.github.com/repos/ssadedin/bpipe/labels/Priority-Medium","name":"Priority-Medium","color":"FFFFFF"}],"state":"closed","locked":false,"assignee":{"login":"ssadedin","id":138868,"avatar_url":"https://avatars.githubusercontent.com/u/138868?v=3","gravatar_id":"","url":"https://api.github.com/users/ssadedin","html_url":"https://github.com/ssadedin","followers_url":"https://api.github.com/users/ssadedin/followers","following_url":"https://api.github.com/users/ssadedin/following{/other_user}","gists_url":"https://api.github.com/users/ssadedin/gists{/gist_id}","starred_url":"https://api.github.com/users/ssadedin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ssadedin/subscriptions","organizations_url":"https://api.github.com/users/ssadedin/orgs","repos_url":"https://api.github.com/users/ssadedin/repos","events_url":"https://api.github.com/users/ssadedin/events{/privacy}","received_events_url":"https://api.github.com/users/ssadedin/received_events","type":"User","site_admin":false},"assignees":[{"login":"ssadedin","id":138868,"avatar_url":"https://avatars.githubusercontent.com/u/138868?v=3","gravatar_id":"","url":"https://api.github.com/users/ssadedin","html_url":"https://github.com/ssadedin","followers_url":"https://api.github.com/users/ssadedin/followers","following_url":"https://api.github.com/users/ssadedin/following{/other_user}","gists_url":"https://api.github.com/users/ssadedin/gists{/gist_id}","starred_url":"https://api.github.com/users/ssadedin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ssadedin/subscriptions","organizations_url":"https://api.github.com/users/ssadedin/orgs","repos_url":"https://api.github.com/users/ssadedin/repos","events_url":"https://api.github.com/users/ssadedin/events{/privacy}","received_events_url":"https://api.github.com/users/ssadedin/received_events","type":"User","site_admin":false}],"milestone":null,"comments":17,"created_at":"2014-12-18T15:35:07Z","updated_at":"2016-06-29T15:37:25Z","closed_at":"2016-06-29T15:37:25Z","body":"I have a bpipe pipeline with 5 stages and 436 input files.\r\n\r\nI can successfully start the pipeline if there are no input files present, and if I leave it running it will complete, but only if I set java memory to 2048.\r\n\r\nIf I have to stop and re-start the pipeline, with the 436 input files and 3384 output files.\r\n\r\nBpipe tries to crunch through and check the files, but eventually hangs with 500-700% CPU, and the pipeline never starts."}}
{"actor":{"display_login":"gdevenyi"}}