{"action":"opened","number":3011,"pull_request":{"url":"https://api.github.com/repos/pymc-devs/pymc3/pulls/3011","id":193335899,"node_id":"MDExOlB1bGxSZXF1ZXN0MTkzMzM1ODk5","html_url":"https://github.com/pymc-devs/pymc3/pull/3011","diff_url":"https://github.com/pymc-devs/pymc3/pull/3011.diff","patch_url":"https://github.com/pymc-devs/pymc3/pull/3011.patch","issue_url":"https://api.github.com/repos/pymc-devs/pymc3/issues/3011","number":3011,"state":"open","locked":false,"title":"Rewrite parallel sampling using multiprocessing","user":{"login":"aseyboldt","id":1882397,"node_id":"MDQ6VXNlcjE4ODIzOTc=","avatar_url":"https://avatars3.githubusercontent.com/u/1882397?v=4","gravatar_id":"","url":"https://api.github.com/users/aseyboldt","html_url":"https://github.com/aseyboldt","followers_url":"https://api.github.com/users/aseyboldt/followers","following_url":"https://api.github.com/users/aseyboldt/following{/other_user}","gists_url":"https://api.github.com/users/aseyboldt/gists{/gist_id}","starred_url":"https://api.github.com/users/aseyboldt/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aseyboldt/subscriptions","organizations_url":"https://api.github.com/users/aseyboldt/orgs","repos_url":"https://api.github.com/users/aseyboldt/repos","events_url":"https://api.github.com/users/aseyboldt/events{/privacy}","received_events_url":"https://api.github.com/users/aseyboldt/received_events","type":"User","site_admin":false},"body":"This PR gets rid of joblib altogether, and replaces it with a custom implementation.\r\nThis should solve some long standing problems:\r\n- Pickling models in not always possible, and with this approach we only need to do this on windows now.\r\n- Problems pickling large traces. Right now we send the finished trace to the main thread by pickling it and sending it though a pipe. This causes problems if the trace is very large, as on many systems there is a limit for the size of one `pipe.send`. Very large traces can't use multiprocessing right now.\r\n- joblib doesn't seem to be particularly reliable. I've seen lots of cases where processes got stuck, and I had to kill them manually, or interrupting sampling works only after sending several interrupts. (One caviat with the proposed method is that we can only interrupt traces between samples. So very slow samplers might be stuck for a few seconds. We can still kill them though.)\r\n- More control over how exceptions are reported. We can format the remote exceptions nicely. This is not implemented yet, but there is code for this here: https://hg.python.org/cpython/rev/c4f92b597074\r\n- Better progress bars, as the main processes always knows how far the individual processes are. (but there seem to be tqdm bugs that still mess that up...)\r\n- We can implement trace backends so that the main process writes all samples to the file as they come in, and we never need to keep many of them in memory, and can avoid parallel io.\r\n\r\n---\r\n\r\nWe start one process per chain, that communicates with the main thread by sending messages through a pipe. Ones a draw is ready, it is stored in shared memory, and the main process can access it. Then the sampler process is asked to write the next sample to that shared memory.\r\n\r\nPerformance wise it shouldn't be much different, the pipes seem to be fast enough to deal with the speed of our sampler. It is possible that we gain a bit of speed for large models with cheap computations, as we have a dedicated thread for sampling, that doesn't also have to deal with storing data in the trace. For very small models it might be a bit slower, as we do have some small constant overhead for sending messages.\r\n\r\n### TODO\r\n- [ ] Add warnings (they are not working yet)\r\n- [ ] Fix progress bar\r\n- [ ] Better formatting of exceptions","created_at":"2018-06-07T14:12:09Z","updated_at":"2018-06-07T14:12:09Z","closed_at":null,"merged_at":null,"merge_commit_sha":null,"assignee":null,"assignees":[],"requested_reviewers":[],"requested_teams":[],"labels":[],"milestone":null,"commits_url":"https://api.github.com/repos/pymc-devs/pymc3/pulls/3011/commits","review_comments_url":"https://api.github.com/repos/pymc-devs/pymc3/pulls/3011/comments","review_comment_url":"https://api.github.com/repos/pymc-devs/pymc3/pulls/comments{/number}","comments_url":"https://api.github.com/repos/pymc-devs/pymc3/issues/3011/comments","statuses_url":"https://api.github.com/repos/pymc-devs/pymc3/statuses/42390664460caed47a38cfa0b54845df3d3fc111","head":{"label":"aseyboldt:multiproc","ref":"multiproc","sha":"42390664460caed47a38cfa0b54845df3d3fc111","user":{"login":"aseyboldt","id":1882397,"node_id":"MDQ6VXNlcjE4ODIzOTc=","avatar_url":"https://avatars3.githubusercontent.com/u/1882397?v=4","gravatar_id":"","url":"https://api.github.com/users/aseyboldt","html_url":"https://github.com/aseyboldt","followers_url":"https://api.github.com/users/aseyboldt/followers","following_url":"https://api.github.com/users/aseyboldt/following{/other_user}","gists_url":"https://api.github.com/users/aseyboldt/gists{/gist_id}","starred_url":"https://api.github.com/users/aseyboldt/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aseyboldt/subscriptions","organizations_url":"https://api.github.com/users/aseyboldt/orgs","repos_url":"https://api.github.com/users/aseyboldt/repos","events_url":"https://api.github.com/users/aseyboldt/events{/privacy}","received_events_url":"https://api.github.com/users/aseyboldt/received_events","type":"User","site_admin":false},"repo":{"id":79482338,"node_id":"MDEwOlJlcG9zaXRvcnk3OTQ4MjMzOA==","name":"pymc3","full_name":"aseyboldt/pymc3","owner":{"login":"aseyboldt","id":1882397,"node_id":"MDQ6VXNlcjE4ODIzOTc=","avatar_url":"https://avatars3.githubusercontent.com/u/1882397?v=4","gravatar_id":"","url":"https://api.github.com/users/aseyboldt","html_url":"https://github.com/aseyboldt","followers_url":"https://api.github.com/users/aseyboldt/followers","following_url":"https://api.github.com/users/aseyboldt/following{/other_user}","gists_url":"https://api.github.com/users/aseyboldt/gists{/gist_id}","starred_url":"https://api.github.com/users/aseyboldt/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aseyboldt/subscriptions","organizations_url":"https://api.github.com/users/aseyboldt/orgs","repos_url":"https://api.github.com/users/aseyboldt/repos","events_url":"https://api.github.com/users/aseyboldt/events{/privacy}","received_events_url":"https://api.github.com/users/aseyboldt/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/aseyboldt/pymc3","description":"Probabilistic Programming in Python: Bayesian Modeling and Probabilistic Machine Learning with Theano","fork":true,"url":"https://api.github.com/repos/aseyboldt/pymc3","forks_url":"https://api.github.com/repos/aseyboldt/pymc3/forks","keys_url":"https://api.github.com/repos/aseyboldt/pymc3/keys{/key_id}","collaborators_url":"https://api.github.com/repos/aseyboldt/pymc3/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/aseyboldt/pymc3/teams","hooks_url":"https://api.github.com/repos/aseyboldt/pymc3/hooks","issue_events_url":"https://api.github.com/repos/aseyboldt/pymc3/issues/events{/number}","events_url":"https://api.github.com/repos/aseyboldt/pymc3/events","assignees_url":"https://api.github.com/repos/aseyboldt/pymc3/assignees{/user}","branches_url":"https://api.github.com/repos/aseyboldt/pymc3/branches{/branch}","tags_url":"https://api.github.com/repos/aseyboldt/pymc3/tags","blobs_url":"https://api.github.com/repos/aseyboldt/pymc3/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/aseyboldt/pymc3/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/aseyboldt/pymc3/git/refs{/sha}","trees_url":"https://api.github.com/repos/aseyboldt/pymc3/git/trees{/sha}","statuses_url":"https://api.github.com/repos/aseyboldt/pymc3/statuses/{sha}","languages_url":"https://api.github.com/repos/aseyboldt/pymc3/languages","stargazers_url":"https://api.github.com/repos/aseyboldt/pymc3/stargazers","contributors_url":"https://api.github.com/repos/aseyboldt/pymc3/contributors","subscribers_url":"https://api.github.com/repos/aseyboldt/pymc3/subscribers","subscription_url":"https://api.github.com/repos/aseyboldt/pymc3/subscription","commits_url":"https://api.github.com/repos/aseyboldt/pymc3/commits{/sha}","git_commits_url":"https://api.github.com/repos/aseyboldt/pymc3/git/commits{/sha}","comments_url":"https://api.github.com/repos/aseyboldt/pymc3/comments{/number}","issue_comment_url":"https://api.github.com/repos/aseyboldt/pymc3/issues/comments{/number}","contents_url":"https://api.github.com/repos/aseyboldt/pymc3/contents/{+path}","compare_url":"https://api.github.com/repos/aseyboldt/pymc3/compare/{base}...{head}","merges_url":"https://api.github.com/repos/aseyboldt/pymc3/merges","archive_url":"https://api.github.com/repos/aseyboldt/pymc3/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/aseyboldt/pymc3/downloads","issues_url":"https://api.github.com/repos/aseyboldt/pymc3/issues{/number}","pulls_url":"https://api.github.com/repos/aseyboldt/pymc3/pulls{/number}","milestones_url":"https://api.github.com/repos/aseyboldt/pymc3/milestones{/number}","notifications_url":"https://api.github.com/repos/aseyboldt/pymc3/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/aseyboldt/pymc3/labels{/name}","releases_url":"https://api.github.com/repos/aseyboldt/pymc3/releases{/id}","deployments_url":"https://api.github.com/repos/aseyboldt/pymc3/deployments","created_at":"2017-01-19T18:30:16Z","updated_at":"2017-01-19T18:30:20Z","pushed_at":"2018-06-07T13:55:39Z","git_url":"git://github.com/aseyboldt/pymc3.git","ssh_url":"git@github.com:aseyboldt/pymc3.git","clone_url":"https://github.com/aseyboldt/pymc3.git","svn_url":"https://github.com/aseyboldt/pymc3","homepage":"http://pymc-devs.github.io/pymc3/","size":265381,"stargazers_count":0,"watchers_count":0,"language":"Python","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"archived":false,"open_issues_count":0,"license":{"key":"other","name":"Other","spdx_id":null,"url":null,"node_id":"MDc6TGljZW5zZTA="},"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"pymc-devs:master","ref":"master","sha":"f361844744c4c44dac3bbec01a07603e539868e7","user":{"login":"pymc-devs","id":81121,"node_id":"MDEyOk9yZ2FuaXphdGlvbjgxMTIx","avatar_url":"https://avatars3.githubusercontent.com/u/81121?v=4","gravatar_id":"","url":"https://api.github.com/users/pymc-devs","html_url":"https://github.com/pymc-devs","followers_url":"https://api.github.com/users/pymc-devs/followers","following_url":"https://api.github.com/users/pymc-devs/following{/other_user}","gists_url":"https://api.github.com/users/pymc-devs/gists{/gist_id}","starred_url":"https://api.github.com/users/pymc-devs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pymc-devs/subscriptions","organizations_url":"https://api.github.com/users/pymc-devs/orgs","repos_url":"https://api.github.com/users/pymc-devs/repos","events_url":"https://api.github.com/users/pymc-devs/events{/privacy}","received_events_url":"https://api.github.com/users/pymc-devs/received_events","type":"Organization","site_admin":false},"repo":{"id":192904,"node_id":"MDEwOlJlcG9zaXRvcnkxOTI5MDQ=","name":"pymc3","full_name":"pymc-devs/pymc3","owner":{"login":"pymc-devs","id":81121,"node_id":"MDEyOk9yZ2FuaXphdGlvbjgxMTIx","avatar_url":"https://avatars3.githubusercontent.com/u/81121?v=4","gravatar_id":"","url":"https://api.github.com/users/pymc-devs","html_url":"https://github.com/pymc-devs","followers_url":"https://api.github.com/users/pymc-devs/followers","following_url":"https://api.github.com/users/pymc-devs/following{/other_user}","gists_url":"https://api.github.com/users/pymc-devs/gists{/gist_id}","starred_url":"https://api.github.com/users/pymc-devs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pymc-devs/subscriptions","organizations_url":"https://api.github.com/users/pymc-devs/orgs","repos_url":"https://api.github.com/users/pymc-devs/repos","events_url":"https://api.github.com/users/pymc-devs/events{/privacy}","received_events_url":"https://api.github.com/users/pymc-devs/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/pymc-devs/pymc3","description":"Probabilistic Programming in Python: Bayesian Modeling and Probabilistic Machine Learning with Theano","fork":false,"url":"https://api.github.com/repos/pymc-devs/pymc3","forks_url":"https://api.github.com/repos/pymc-devs/pymc3/forks","keys_url":"https://api.github.com/repos/pymc-devs/pymc3/keys{/key_id}","collaborators_url":"https://api.github.com/repos/pymc-devs/pymc3/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/pymc-devs/pymc3/teams","hooks_url":"https://api.github.com/repos/pymc-devs/pymc3/hooks","issue_events_url":"https://api.github.com/repos/pymc-devs/pymc3/issues/events{/number}","events_url":"https://api.github.com/repos/pymc-devs/pymc3/events","assignees_url":"https://api.github.com/repos/pymc-devs/pymc3/assignees{/user}","branches_url":"https://api.github.com/repos/pymc-devs/pymc3/branches{/branch}","tags_url":"https://api.github.com/repos/pymc-devs/pymc3/tags","blobs_url":"https://api.github.com/repos/pymc-devs/pymc3/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/pymc-devs/pymc3/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/pymc-devs/pymc3/git/refs{/sha}","trees_url":"https://api.github.com/repos/pymc-devs/pymc3/git/trees{/sha}","statuses_url":"https://api.github.com/repos/pymc-devs/pymc3/statuses/{sha}","languages_url":"https://api.github.com/repos/pymc-devs/pymc3/languages","stargazers_url":"https://api.github.com/repos/pymc-devs/pymc3/stargazers","contributors_url":"https://api.github.com/repos/pymc-devs/pymc3/contributors","subscribers_url":"https://api.github.com/repos/pymc-devs/pymc3/subscribers","subscription_url":"https://api.github.com/repos/pymc-devs/pymc3/subscription","commits_url":"https://api.github.com/repos/pymc-devs/pymc3/commits{/sha}","git_commits_url":"https://api.github.com/repos/pymc-devs/pymc3/git/commits{/sha}","comments_url":"https://api.github.com/repos/pymc-devs/pymc3/comments{/number}","issue_comment_url":"https://api.github.com/repos/pymc-devs/pymc3/issues/comments{/number}","contents_url":"https://api.github.com/repos/pymc-devs/pymc3/contents/{+path}","compare_url":"https://api.github.com/repos/pymc-devs/pymc3/compare/{base}...{head}","merges_url":"https://api.github.com/repos/pymc-devs/pymc3/merges","archive_url":"https://api.github.com/repos/pymc-devs/pymc3/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/pymc-devs/pymc3/downloads","issues_url":"https://api.github.com/repos/pymc-devs/pymc3/issues{/number}","pulls_url":"https://api.github.com/repos/pymc-devs/pymc3/pulls{/number}","milestones_url":"https://api.github.com/repos/pymc-devs/pymc3/milestones{/number}","notifications_url":"https://api.github.com/repos/pymc-devs/pymc3/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/pymc-devs/pymc3/labels{/name}","releases_url":"https://api.github.com/repos/pymc-devs/pymc3/releases{/id}","deployments_url":"https://api.github.com/repos/pymc-devs/pymc3/deployments","created_at":"2009-05-05T09:43:50Z","updated_at":"2018-06-07T14:06:18Z","pushed_at":"2018-06-07T07:52:49Z","git_url":"git://github.com/pymc-devs/pymc3.git","ssh_url":"git@github.com:pymc-devs/pymc3.git","clone_url":"https://github.com/pymc-devs/pymc3.git","svn_url":"https://github.com/pymc-devs/pymc3","homepage":"http://pymc-devs.github.io/pymc3/","size":229641,"stargazers_count":3322,"watchers_count":3322,"language":"Python","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":823,"mirror_url":null,"archived":false,"open_issues_count":252,"license":{"key":"other","name":"Other","spdx_id":null,"url":null,"node_id":"MDc6TGljZW5zZTA="},"forks":823,"open_issues":252,"watchers":3322,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/pymc-devs/pymc3/pulls/3011"},"html":{"href":"https://github.com/pymc-devs/pymc3/pull/3011"},"issue":{"href":"https://api.github.com/repos/pymc-devs/pymc3/issues/3011"},"comments":{"href":"https://api.github.com/repos/pymc-devs/pymc3/issues/3011/comments"},"review_comments":{"href":"https://api.github.com/repos/pymc-devs/pymc3/pulls/3011/comments"},"review_comment":{"href":"https://api.github.com/repos/pymc-devs/pymc3/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/pymc-devs/pymc3/pulls/3011/commits"},"statuses":{"href":"https://api.github.com/repos/pymc-devs/pymc3/statuses/42390664460caed47a38cfa0b54845df3d3fc111"}},"author_association":"MEMBER","merged":false,"mergeable":null,"rebaseable":null,"mergeable_state":"unknown","merged_by":null,"comments":0,"review_comments":0,"maintainer_can_modify":true,"commits":2,"additions":440,"deletions":15,"changed_files":5}}
{"actor":{"display_login":"aseyboldt"}}