{"action":"closed","number":70,"pull_request":{"url":"https://api.github.com/repos/simonsfoundation/inferelator_ng/pulls/70","id":212446272,"node_id":"MDExOlB1bGxSZXF1ZXN0MjEyNDQ2Mjcy","html_url":"https://github.com/simonsfoundation/inferelator_ng/pull/70","diff_url":"https://github.com/simonsfoundation/inferelator_ng/pull/70.diff","patch_url":"https://github.com/simonsfoundation/inferelator_ng/pull/70.patch","issue_url":"https://api.github.com/repos/simonsfoundation/inferelator_ng/issues/70","number":70,"state":"closed","locked":false,"title":"Memory optimization","user":{"login":"asistradition","id":33259488,"node_id":"MDQ6VXNlcjMzMjU5NDg4","avatar_url":"https://avatars3.githubusercontent.com/u/33259488?v=4","gravatar_id":"","url":"https://api.github.com/users/asistradition","html_url":"https://github.com/asistradition","followers_url":"https://api.github.com/users/asistradition/followers","following_url":"https://api.github.com/users/asistradition/following{/other_user}","gists_url":"https://api.github.com/users/asistradition/gists{/gist_id}","starred_url":"https://api.github.com/users/asistradition/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/asistradition/subscriptions","organizations_url":"https://api.github.com/users/asistradition/orgs","repos_url":"https://api.github.com/users/asistradition/repos","events_url":"https://api.github.com/users/asistradition/events{/privacy}","received_events_url":"https://api.github.com/users/asistradition/received_events","type":"User","site_admin":false},"body":"Aight this is the rewritten code that we were discussing today. It should be mostly a drop in replacement for the older code. There are a ton of changes, but they're all pretty straightforward. The regression engine is faster but functionally results in the same output. The MI engine is entirely python with standard dependencies, and returns mostly the same output (to float tolerance). The exception is the CLR change that was discussed today. \r\n\r\nThe workflow is lighter in memory usage; after it finishes setup, workflow.design and workflow.response should be the only heavy memory objects remaining (the rest should dereferenced for garbage collection). To mitigate problems with memory spiking during setup, workflow.async_start can be set True, and the code in startup_run will run on workflow.async_chunk number of processes. Once one process finishes startup_run, another will start, until everyone's finished. It'll then run startup_final with all the processes and move on to running the regression. MI and the regression driver do copy workflow.design and workflow.response, but not at the same time, and those copies are dereferenced for garbage collection. Peak memory usage during regression should be a little over 4 times the expression data once it's in an array, plus overhead from the interpreter. I believe that all KVS objects are torn down immediately after all of the client processes have accessed them. \r\n\r\nThe utils package has a Debug class that's just used in place of print and can be set to varying levels of verbosity. It also has a kvs_async class that has process control methods to manage staging through KVS. \r\n\r\nThe entire thing runs parallelized with python 2.7. It runs nicely in 3.6 as well, provided that there is only one process. There is some desync bug in 3.6 that turns the output to garbage with multiple processes (it fails silently, so if you run this on the HPC with 3.6 it'll just give you nonsense). It passes all of the existing tests, but I haven't implemented any new ones and it may be worth comparing your specific output before & after this branch.","created_at":"2018-08-31T17:20:32Z","updated_at":"2018-11-19T18:11:53Z","closed_at":"2018-11-19T18:11:53Z","merged_at":null,"merge_commit_sha":"92fb0bd2713b7cd693745813f2065c988b4fd50e","assignee":null,"assignees":[],"requested_reviewers":[],"requested_teams":[],"labels":[],"milestone":null,"commits_url":"https://api.github.com/repos/simonsfoundation/inferelator_ng/pulls/70/commits","review_comments_url":"https://api.github.com/repos/simonsfoundation/inferelator_ng/pulls/70/comments","review_comment_url":"https://api.github.com/repos/simonsfoundation/inferelator_ng/pulls/comments{/number}","comments_url":"https://api.github.com/repos/simonsfoundation/inferelator_ng/issues/70/comments","statuses_url":"https://api.github.com/repos/simonsfoundation/inferelator_ng/statuses/32ace98728bf7ee709d3d74dded2b8d366b105a8","head":{"label":"asistradition:memory_optimization","ref":"memory_optimization","sha":"32ace98728bf7ee709d3d74dded2b8d366b105a8","user":{"login":"asistradition","id":33259488,"node_id":"MDQ6VXNlcjMzMjU5NDg4","avatar_url":"https://avatars3.githubusercontent.com/u/33259488?v=4","gravatar_id":"","url":"https://api.github.com/users/asistradition","html_url":"https://github.com/asistradition","followers_url":"https://api.github.com/users/asistradition/followers","following_url":"https://api.github.com/users/asistradition/following{/other_user}","gists_url":"https://api.github.com/users/asistradition/gists{/gist_id}","starred_url":"https://api.github.com/users/asistradition/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/asistradition/subscriptions","organizations_url":"https://api.github.com/users/asistradition/orgs","repos_url":"https://api.github.com/users/asistradition/repos","events_url":"https://api.github.com/users/asistradition/events{/privacy}","received_events_url":"https://api.github.com/users/asistradition/received_events","type":"User","site_admin":false},"repo":{"id":136524376,"node_id":"MDEwOlJlcG9zaXRvcnkxMzY1MjQzNzY=","name":"inferelator_ng","full_name":"asistradition/inferelator_ng","private":false,"owner":{"login":"asistradition","id":33259488,"node_id":"MDQ6VXNlcjMzMjU5NDg4","avatar_url":"https://avatars3.githubusercontent.com/u/33259488?v=4","gravatar_id":"","url":"https://api.github.com/users/asistradition","html_url":"https://github.com/asistradition","followers_url":"https://api.github.com/users/asistradition/followers","following_url":"https://api.github.com/users/asistradition/following{/other_user}","gists_url":"https://api.github.com/users/asistradition/gists{/gist_id}","starred_url":"https://api.github.com/users/asistradition/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/asistradition/subscriptions","organizations_url":"https://api.github.com/users/asistradition/orgs","repos_url":"https://api.github.com/users/asistradition/repos","events_url":"https://api.github.com/users/asistradition/events{/privacy}","received_events_url":"https://api.github.com/users/asistradition/received_events","type":"User","site_admin":false},"html_url":"https://github.com/asistradition/inferelator_ng","description":"Inferelator refactor and generalization.","fork":true,"url":"https://api.github.com/repos/asistradition/inferelator_ng","forks_url":"https://api.github.com/repos/asistradition/inferelator_ng/forks","keys_url":"https://api.github.com/repos/asistradition/inferelator_ng/keys{/key_id}","collaborators_url":"https://api.github.com/repos/asistradition/inferelator_ng/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/asistradition/inferelator_ng/teams","hooks_url":"https://api.github.com/repos/asistradition/inferelator_ng/hooks","issue_events_url":"https://api.github.com/repos/asistradition/inferelator_ng/issues/events{/number}","events_url":"https://api.github.com/repos/asistradition/inferelator_ng/events","assignees_url":"https://api.github.com/repos/asistradition/inferelator_ng/assignees{/user}","branches_url":"https://api.github.com/repos/asistradition/inferelator_ng/branches{/branch}","tags_url":"https://api.github.com/repos/asistradition/inferelator_ng/tags","blobs_url":"https://api.github.com/repos/asistradition/inferelator_ng/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/asistradition/inferelator_ng/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/asistradition/inferelator_ng/git/refs{/sha}","trees_url":"https://api.github.com/repos/asistradition/inferelator_ng/git/trees{/sha}","statuses_url":"https://api.github.com/repos/asistradition/inferelator_ng/statuses/{sha}","languages_url":"https://api.github.com/repos/asistradition/inferelator_ng/languages","stargazers_url":"https://api.github.com/repos/asistradition/inferelator_ng/stargazers","contributors_url":"https://api.github.com/repos/asistradition/inferelator_ng/contributors","subscribers_url":"https://api.github.com/repos/asistradition/inferelator_ng/subscribers","subscription_url":"https://api.github.com/repos/asistradition/inferelator_ng/subscription","commits_url":"https://api.github.com/repos/asistradition/inferelator_ng/commits{/sha}","git_commits_url":"https://api.github.com/repos/asistradition/inferelator_ng/git/commits{/sha}","comments_url":"https://api.github.com/repos/asistradition/inferelator_ng/comments{/number}","issue_comment_url":"https://api.github.com/repos/asistradition/inferelator_ng/issues/comments{/number}","contents_url":"https://api.github.com/repos/asistradition/inferelator_ng/contents/{+path}","compare_url":"https://api.github.com/repos/asistradition/inferelator_ng/compare/{base}...{head}","merges_url":"https://api.github.com/repos/asistradition/inferelator_ng/merges","archive_url":"https://api.github.com/repos/asistradition/inferelator_ng/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/asistradition/inferelator_ng/downloads","issues_url":"https://api.github.com/repos/asistradition/inferelator_ng/issues{/number}","pulls_url":"https://api.github.com/repos/asistradition/inferelator_ng/pulls{/number}","milestones_url":"https://api.github.com/repos/asistradition/inferelator_ng/milestones{/number}","notifications_url":"https://api.github.com/repos/asistradition/inferelator_ng/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/asistradition/inferelator_ng/labels{/name}","releases_url":"https://api.github.com/repos/asistradition/inferelator_ng/releases{/id}","deployments_url":"https://api.github.com/repos/asistradition/inferelator_ng/deployments","created_at":"2018-06-07T19:53:52Z","updated_at":"2018-11-15T19:49:30Z","pushed_at":"2018-11-19T18:09:53Z","git_url":"git://github.com/asistradition/inferelator_ng.git","ssh_url":"git@github.com:asistradition/inferelator_ng.git","clone_url":"https://github.com/asistradition/inferelator_ng.git","svn_url":"https://github.com/asistradition/inferelator_ng","homepage":"","size":15628,"stargazers_count":0,"watchers_count":0,"language":"Python","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"archived":false,"open_issues_count":0,"license":{"key":"bsd-2-clause","name":"BSD 2-Clause \"Simplified\" License","spdx_id":"BSD-2-Clause","url":"https://api.github.com/licenses/bsd-2-clause","node_id":"MDc6TGljZW5zZTQ="},"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"simonsfoundation:master","ref":"master","sha":"cdb1b39d2a7ffc162aec0099f2b30ae37de19e61","user":{"login":"simonsfoundation","id":7658073,"node_id":"MDEyOk9yZ2FuaXphdGlvbjc2NTgwNzM=","avatar_url":"https://avatars1.githubusercontent.com/u/7658073?v=4","gravatar_id":"","url":"https://api.github.com/users/simonsfoundation","html_url":"https://github.com/simonsfoundation","followers_url":"https://api.github.com/users/simonsfoundation/followers","following_url":"https://api.github.com/users/simonsfoundation/following{/other_user}","gists_url":"https://api.github.com/users/simonsfoundation/gists{/gist_id}","starred_url":"https://api.github.com/users/simonsfoundation/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/simonsfoundation/subscriptions","organizations_url":"https://api.github.com/users/simonsfoundation/orgs","repos_url":"https://api.github.com/users/simonsfoundation/repos","events_url":"https://api.github.com/users/simonsfoundation/events{/privacy}","received_events_url":"https://api.github.com/users/simonsfoundation/received_events","type":"Organization","site_admin":false},"repo":{"id":61822570,"node_id":"MDEwOlJlcG9zaXRvcnk2MTgyMjU3MA==","name":"inferelator_ng","full_name":"simonsfoundation/inferelator_ng","private":false,"owner":{"login":"simonsfoundation","id":7658073,"node_id":"MDEyOk9yZ2FuaXphdGlvbjc2NTgwNzM=","avatar_url":"https://avatars1.githubusercontent.com/u/7658073?v=4","gravatar_id":"","url":"https://api.github.com/users/simonsfoundation","html_url":"https://github.com/simonsfoundation","followers_url":"https://api.github.com/users/simonsfoundation/followers","following_url":"https://api.github.com/users/simonsfoundation/following{/other_user}","gists_url":"https://api.github.com/users/simonsfoundation/gists{/gist_id}","starred_url":"https://api.github.com/users/simonsfoundation/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/simonsfoundation/subscriptions","organizations_url":"https://api.github.com/users/simonsfoundation/orgs","repos_url":"https://api.github.com/users/simonsfoundation/repos","events_url":"https://api.github.com/users/simonsfoundation/events{/privacy}","received_events_url":"https://api.github.com/users/simonsfoundation/received_events","type":"Organization","site_admin":false},"html_url":"https://github.com/simonsfoundation/inferelator_ng","description":"Inferelator refactor and generalization.","fork":false,"url":"https://api.github.com/repos/simonsfoundation/inferelator_ng","forks_url":"https://api.github.com/repos/simonsfoundation/inferelator_ng/forks","keys_url":"https://api.github.com/repos/simonsfoundation/inferelator_ng/keys{/key_id}","collaborators_url":"https://api.github.com/repos/simonsfoundation/inferelator_ng/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/simonsfoundation/inferelator_ng/teams","hooks_url":"https://api.github.com/repos/simonsfoundation/inferelator_ng/hooks","issue_events_url":"https://api.github.com/repos/simonsfoundation/inferelator_ng/issues/events{/number}","events_url":"https://api.github.com/repos/simonsfoundation/inferelator_ng/events","assignees_url":"https://api.github.com/repos/simonsfoundation/inferelator_ng/assignees{/user}","branches_url":"https://api.github.com/repos/simonsfoundation/inferelator_ng/branches{/branch}","tags_url":"https://api.github.com/repos/simonsfoundation/inferelator_ng/tags","blobs_url":"https://api.github.com/repos/simonsfoundation/inferelator_ng/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/simonsfoundation/inferelator_ng/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/simonsfoundation/inferelator_ng/git/refs{/sha}","trees_url":"https://api.github.com/repos/simonsfoundation/inferelator_ng/git/trees{/sha}","statuses_url":"https://api.github.com/repos/simonsfoundation/inferelator_ng/statuses/{sha}","languages_url":"https://api.github.com/repos/simonsfoundation/inferelator_ng/languages","stargazers_url":"https://api.github.com/repos/simonsfoundation/inferelator_ng/stargazers","contributors_url":"https://api.github.com/repos/simonsfoundation/inferelator_ng/contributors","subscribers_url":"https://api.github.com/repos/simonsfoundation/inferelator_ng/subscribers","subscription_url":"https://api.github.com/repos/simonsfoundation/inferelator_ng/subscription","commits_url":"https://api.github.com/repos/simonsfoundation/inferelator_ng/commits{/sha}","git_commits_url":"https://api.github.com/repos/simonsfoundation/inferelator_ng/git/commits{/sha}","comments_url":"https://api.github.com/repos/simonsfoundation/inferelator_ng/comments{/number}","issue_comment_url":"https://api.github.com/repos/simonsfoundation/inferelator_ng/issues/comments{/number}","contents_url":"https://api.github.com/repos/simonsfoundation/inferelator_ng/contents/{+path}","compare_url":"https://api.github.com/repos/simonsfoundation/inferelator_ng/compare/{base}...{head}","merges_url":"https://api.github.com/repos/simonsfoundation/inferelator_ng/merges","archive_url":"https://api.github.com/repos/simonsfoundation/inferelator_ng/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/simonsfoundation/inferelator_ng/downloads","issues_url":"https://api.github.com/repos/simonsfoundation/inferelator_ng/issues{/number}","pulls_url":"https://api.github.com/repos/simonsfoundation/inferelator_ng/pulls{/number}","milestones_url":"https://api.github.com/repos/simonsfoundation/inferelator_ng/milestones{/number}","notifications_url":"https://api.github.com/repos/simonsfoundation/inferelator_ng/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/simonsfoundation/inferelator_ng/labels{/name}","releases_url":"https://api.github.com/repos/simonsfoundation/inferelator_ng/releases{/id}","deployments_url":"https://api.github.com/repos/simonsfoundation/inferelator_ng/deployments","created_at":"2016-06-23T17:05:23Z","updated_at":"2018-09-04T11:59:54Z","pushed_at":"2018-09-11T14:03:04Z","git_url":"git://github.com/simonsfoundation/inferelator_ng.git","ssh_url":"git@github.com:simonsfoundation/inferelator_ng.git","clone_url":"https://github.com/simonsfoundation/inferelator_ng.git","svn_url":"https://github.com/simonsfoundation/inferelator_ng","homepage":"","size":20525,"stargazers_count":6,"watchers_count":6,"language":"Python","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":13,"mirror_url":null,"archived":false,"open_issues_count":16,"license":{"key":"bsd-2-clause","name":"BSD 2-Clause \"Simplified\" License","spdx_id":"BSD-2-Clause","url":"https://api.github.com/licenses/bsd-2-clause","node_id":"MDc6TGljZW5zZTQ="},"forks":13,"open_issues":16,"watchers":6,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/simonsfoundation/inferelator_ng/pulls/70"},"html":{"href":"https://github.com/simonsfoundation/inferelator_ng/pull/70"},"issue":{"href":"https://api.github.com/repos/simonsfoundation/inferelator_ng/issues/70"},"comments":{"href":"https://api.github.com/repos/simonsfoundation/inferelator_ng/issues/70/comments"},"review_comments":{"href":"https://api.github.com/repos/simonsfoundation/inferelator_ng/pulls/70/comments"},"review_comment":{"href":"https://api.github.com/repos/simonsfoundation/inferelator_ng/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/simonsfoundation/inferelator_ng/pulls/70/commits"},"statuses":{"href":"https://api.github.com/repos/simonsfoundation/inferelator_ng/statuses/32ace98728bf7ee709d3d74dded2b8d366b105a8"}},"author_association":"NONE","merged":false,"mergeable":false,"rebaseable":false,"mergeable_state":"dirty","merged_by":null,"comments":2,"review_comments":0,"maintainer_can_modify":false,"commits":128,"additions":1334,"deletions":649,"changed_files":22}}
{"actor":{"display_login":"asistradition"}}