{"action":"created","issue":{"url":"https://api.github.com/repos/LLNL/lbann/issues/256","repository_url":"https://api.github.com/repos/LLNL/lbann","labels_url":"https://api.github.com/repos/LLNL/lbann/issues/256/labels{/name}","comments_url":"https://api.github.com/repos/LLNL/lbann/issues/256/comments","events_url":"https://api.github.com/repos/LLNL/lbann/issues/256/events","html_url":"https://github.com/LLNL/lbann/pull/256","id":307862980,"number":256,"title":"Debugging LTFB","user":{"login":"timmoon10","id":4406448,"avatar_url":"https://avatars0.githubusercontent.com/u/4406448?v=4","gravatar_id":"","url":"https://api.github.com/users/timmoon10","html_url":"https://github.com/timmoon10","followers_url":"https://api.github.com/users/timmoon10/followers","following_url":"https://api.github.com/users/timmoon10/following{/other_user}","gists_url":"https://api.github.com/users/timmoon10/gists{/gist_id}","starred_url":"https://api.github.com/users/timmoon10/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/timmoon10/subscriptions","organizations_url":"https://api.github.com/users/timmoon10/orgs","repos_url":"https://api.github.com/users/timmoon10/repos","events_url":"https://api.github.com/users/timmoon10/events{/privacy}","received_events_url":"https://api.github.com/users/timmoon10/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2018-03-23T00:44:44Z","updated_at":"2018-03-26T22:31:06Z","closed_at":"2018-03-23T17:53:45Z","author_association":"COLLABORATOR","pull_request":{"url":"https://api.github.com/repos/LLNL/lbann/pulls/256","html_url":"https://github.com/LLNL/lbann/pull/256","diff_url":"https://github.com/LLNL/lbann/pull/256.diff","patch_url":"https://github.com/LLNL/lbann/pull/256.patch"},"body":"This fixes several problems I found in the LTFB callback:\r\n- We used to get an infinite recursion under certain circumstances (models contain callbacks and the LTFB callback contains a model). To avoid this, LTFB maintains a list of weights rather than a model instance. We'll need to think of a workaround when we want LTFB with multiple model types.\r\n- LTFB correctly resets the execution mode after evaluating the model on the validation set.\r\n- The weights class synchronizes GPUs after CPU-GPU memory transfers.\r\n- The effective mini-batch is only equal to the global mini-batch size when inter-model communication is enabled.\r\n- We now throw an error if inter-model communication and LTFB are enabled at the same time.\r\n\r\nLTFB on LeNet and AlexNet now appears to run reasonably."},"comment":{"url":"https://api.github.com/repos/LLNL/lbann/issues/comments/376333021","html_url":"https://github.com/LLNL/lbann/pull/256#issuecomment-376333021","issue_url":"https://api.github.com/repos/LLNL/lbann/issues/256","id":376333021,"user":{"login":"samadejacobs","id":16696152,"avatar_url":"https://avatars0.githubusercontent.com/u/16696152?v=4","gravatar_id":"","url":"https://api.github.com/users/samadejacobs","html_url":"https://github.com/samadejacobs","followers_url":"https://api.github.com/users/samadejacobs/followers","following_url":"https://api.github.com/users/samadejacobs/following{/other_user}","gists_url":"https://api.github.com/users/samadejacobs/gists{/gist_id}","starred_url":"https://api.github.com/users/samadejacobs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/samadejacobs/subscriptions","organizations_url":"https://api.github.com/users/samadejacobs/orgs","repos_url":"https://api.github.com/users/samadejacobs/repos","events_url":"https://api.github.com/users/samadejacobs/events{/privacy}","received_events_url":"https://api.github.com/users/samadejacobs/received_events","type":"User","site_admin":false},"created_at":"2018-03-26T22:31:06Z","updated_at":"2018-03-26T22:31:06Z","author_association":"COLLABORATOR","body":"The issue reported was fixed by setting data_set_per_model=true in input layer, thanks @bvanessen "}}
{"actor":{"display_login":"samadejacobs"}}