{"action":"created","issue":{"url":"https://api.github.com/repos/BVLC/caffe/issues/3548","labels_url":"https://api.github.com/repos/BVLC/caffe/issues/3548/labels{/name}","comments_url":"https://api.github.com/repos/BVLC/caffe/issues/3548/comments","events_url":"https://api.github.com/repos/BVLC/caffe/issues/3548/events","html_url":"https://github.com/BVLC/caffe/pull/3548","id":126458716,"number":3548,"title":"Python implementation of the orthonormal and LSUV initializations","user":{"login":"ducha-aiki","id":4803565,"avatar_url":"https://avatars.githubusercontent.com/u/4803565?v=3","gravatar_id":"","url":"https://api.github.com/users/ducha-aiki","html_url":"https://github.com/ducha-aiki","followers_url":"https://api.github.com/users/ducha-aiki/followers","following_url":"https://api.github.com/users/ducha-aiki/following{/other_user}","gists_url":"https://api.github.com/users/ducha-aiki/gists{/gist_id}","starred_url":"https://api.github.com/users/ducha-aiki/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ducha-aiki/subscriptions","organizations_url":"https://api.github.com/users/ducha-aiki/orgs","repos_url":"https://api.github.com/users/ducha-aiki/repos","events_url":"https://api.github.com/users/ducha-aiki/events{/privacy}","received_events_url":"https://api.github.com/users/ducha-aiki/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2016-01-13T16:33:10Z","updated_at":"2016-01-13T16:34:24Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/BVLC/caffe/pulls/3548","html_url":"https://github.com/BVLC/caffe/pull/3548","diff_url":"https://github.com/BVLC/caffe/pull/3548.diff","patch_url":"https://github.com/BVLC/caffe/pull/3548.patch"},"body":"LSUV initialization is described in ICLR2016 submission http://arxiv.org/abs/1511.06422\r\nFirst, it fills weight by orthonormal matrices (as suggested by Saxe), 2nd - scale weights using data, until activations vars become 1.0. \r\nIt gives state-of-art or better results for all tested activations, i.e. improving BVLC-googlenet result by 0.8% top-1 accuracy.\r\n\r\nOrthonormal init is described in\r\nAndrew M. Saxe, James L. McClelland, Surya Ganguli (2013). Exact solutions to the nonlinear dynamics of learning in deep linear neural networks.  http://arxiv.org/abs/1312.6120 and is used by Kaggle winners in NDSB (Plankton) and Retinopathy challenges. "},"comment":{"url":"https://api.github.com/repos/BVLC/caffe/issues/comments/171354113","html_url":"https://github.com/BVLC/caffe/pull/3548#issuecomment-171354113","issue_url":"https://api.github.com/repos/BVLC/caffe/issues/3548","id":171354113,"user":{"login":"ducha-aiki","id":4803565,"avatar_url":"https://avatars.githubusercontent.com/u/4803565?v=3","gravatar_id":"","url":"https://api.github.com/users/ducha-aiki","html_url":"https://github.com/ducha-aiki","followers_url":"https://api.github.com/users/ducha-aiki/followers","following_url":"https://api.github.com/users/ducha-aiki/following{/other_user}","gists_url":"https://api.github.com/users/ducha-aiki/gists{/gist_id}","starred_url":"https://api.github.com/users/ducha-aiki/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ducha-aiki/subscriptions","organizations_url":"https://api.github.com/users/ducha-aiki/orgs","repos_url":"https://api.github.com/users/ducha-aiki/repos","events_url":"https://api.github.com/users/ducha-aiki/events{/privacy}","received_events_url":"https://api.github.com/users/ducha-aiki/received_events","type":"User","site_admin":false},"created_at":"2016-01-13T16:34:24Z","updated_at":"2016-01-13T16:34:24Z","body":"Ready to review."}}