{"action":"created","issue":{"url":"https://api.github.com/repos/Lasagne/Lasagne/issues/467","labels_url":"https://api.github.com/repos/Lasagne/Lasagne/issues/467/labels{/name}","comments_url":"https://api.github.com/repos/Lasagne/Lasagne/issues/467/comments","events_url":"https://api.github.com/repos/Lasagne/Lasagne/issues/467/events","html_url":"https://github.com/Lasagne/Lasagne/pull/467","id":111873349,"number":467,"title":"Add BatchNormLayer","user":{"login":"f0k","id":629706,"avatar_url":"https://avatars.githubusercontent.com/u/629706?v=3","gravatar_id":"","url":"https://api.github.com/users/f0k","html_url":"https://github.com/f0k","followers_url":"https://api.github.com/users/f0k/followers","following_url":"https://api.github.com/users/f0k/following{/other_user}","gists_url":"https://api.github.com/users/f0k/gists{/gist_id}","starred_url":"https://api.github.com/users/f0k/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/f0k/subscriptions","organizations_url":"https://api.github.com/users/f0k/orgs","repos_url":"https://api.github.com/users/f0k/repos","events_url":"https://api.github.com/users/f0k/events{/privacy}","received_events_url":"https://api.github.com/users/f0k/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":45,"created_at":"2015-10-16T17:14:12Z","updated_at":"2016-01-04T11:48:30Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/Lasagne/Lasagne/pulls/467","html_url":"https://github.com/Lasagne/Lasagne/pull/467","diff_url":"https://github.com/Lasagne/Lasagne/pull/467.diff","patch_url":"https://github.com/Lasagne/Lasagne/pull/467.patch"},"body":"Finally started integrating my batch normalization implementation in Lasagne. Changes from [the implementation in my gist](https://gist.github.com/f0k/f1a6bd3c8585c400c190):\r\n* Adding epsilon before taking the square root of the variance\r\n* Change parameters to have reduced dimensionality rather than broadcastable singleton dimensions, to match with `BiasLayer` (which in turn was made to match `DenseLayer`, `Conv2DLayer` etc.)\r\n\r\nTODO:\r\n* [x] add tests\r\n* [x] fix LocalResponseNormalization2DLayer docstring while we're at it\r\n\r\nResolves #141."},"comment":{"url":"https://api.github.com/repos/Lasagne/Lasagne/issues/comments/168655094","html_url":"https://github.com/Lasagne/Lasagne/pull/467#issuecomment-168655094","issue_url":"https://api.github.com/repos/Lasagne/Lasagne/issues/467","id":168655094,"user":{"login":"f0k","id":629706,"avatar_url":"https://avatars.githubusercontent.com/u/629706?v=3","gravatar_id":"","url":"https://api.github.com/users/f0k","html_url":"https://github.com/f0k","followers_url":"https://api.github.com/users/f0k/followers","following_url":"https://api.github.com/users/f0k/following{/other_user}","gists_url":"https://api.github.com/users/f0k/gists{/gist_id}","starred_url":"https://api.github.com/users/f0k/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/f0k/subscriptions","organizations_url":"https://api.github.com/users/f0k/orgs","repos_url":"https://api.github.com/users/f0k/repos","events_url":"https://api.github.com/users/f0k/events{/privacy}","received_events_url":"https://api.github.com/users/f0k/received_events","type":"User","site_admin":false},"created_at":"2016-01-04T11:48:30Z","updated_at":"2016-01-04T11:48:30Z","body":"> Right, the \"either std or inv_std is 1\" stuff is a bit tricky I guess.\r\n\r\nNote that this is not the only possible solution, we could also go for `inv_std` completely and not use `T.nnet.batch_normalization()` any more. I just figured we shouldn't drop this possible advantage.\r\n\r\n> Is the original implementation closer to what's described in the paper (or to what the authors of the paper did originally, in case they didn't mention this detail)?\r\n\r\nYes, in the paper it seems they always work with the variance, not the inverse standard deviation. And in the initial cuDNN v4 implementation they also used the variance, but switched to the inverse standard deviation later (this was (and possibly is) still visible from some inconsistencies in the documentation of the release candidate). It's not clear why, maybe it just was to improve performance for the inference step."}}