{"action":"opened","issue":{"url":"https://api.github.com/repos/OpenMined/Grid/issues/49","repository_url":"https://api.github.com/repos/OpenMined/Grid","labels_url":"https://api.github.com/repos/OpenMined/Grid/issues/49/labels{/name}","comments_url":"https://api.github.com/repos/OpenMined/Grid/issues/49/comments","events_url":"https://api.github.com/repos/OpenMined/Grid/issues/49/events","html_url":"https://github.com/OpenMined/Grid/issues/49","id":297907944,"number":49,"title":"Better heuristics for when nodes should give up training","user":{"login":"bendecoste","id":1411145,"avatar_url":"https://avatars2.githubusercontent.com/u/1411145?v=4","gravatar_id":"","url":"https://api.github.com/users/bendecoste","html_url":"https://github.com/bendecoste","followers_url":"https://api.github.com/users/bendecoste/followers","following_url":"https://api.github.com/users/bendecoste/following{/other_user}","gists_url":"https://api.github.com/users/bendecoste/gists{/gist_id}","starred_url":"https://api.github.com/users/bendecoste/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bendecoste/subscriptions","organizations_url":"https://api.github.com/users/bendecoste/orgs","repos_url":"https://api.github.com/users/bendecoste/repos","events_url":"https://api.github.com/users/bendecoste/events{/privacy}","received_events_url":"https://api.github.com/users/bendecoste/received_events","type":"User","site_admin":false},"labels":[{"id":839490306,"url":"https://api.github.com/repos/OpenMined/Grid/labels/TREE","name":"TREE","color":"dafca6","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-02-16T20:18:39Z","updated_at":"2018-02-16T20:18:39Z","closed_at":null,"author_association":"MEMBER","body":"In the MNIST demo, a common problem happens when running with 3 nodes, which is that one of the nodes starts to be unable to positively contribute to training.\r\n\r\nWorker A and worker B start lowering the loss together, and they send their model to worker C, who's data makes the loss go up, so he doesn't publish.\r\n\r\nIf worker C ran more epochs or changed his training rates, he'd probably be able to lower the error rate.\r\n\r\nBasically, the current scheme to decide if a model has improved is very naive.  We should discuss how to make it better!"}}
{"actor":{"display_login":"bendecoste"}}