{"action":"created","issue":{"url":"https://api.github.com/repos/glinscott/leela-chess/issues/199","repository_url":"https://api.github.com/repos/glinscott/leela-chess","labels_url":"https://api.github.com/repos/glinscott/leela-chess/issues/199/labels{/name}","comments_url":"https://api.github.com/repos/glinscott/leela-chess/issues/199/comments","events_url":"https://api.github.com/repos/glinscott/leela-chess/issues/199/events","html_url":"https://github.com/glinscott/leela-chess/issues/199","id":309493051,"number":199,"title":"Weird weights in input convolution","user":{"login":"Ttl","id":544240,"avatar_url":"https://avatars2.githubusercontent.com/u/544240?v=4","gravatar_id":"","url":"https://api.github.com/users/Ttl","html_url":"https://github.com/Ttl","followers_url":"https://api.github.com/users/Ttl/followers","following_url":"https://api.github.com/users/Ttl/following{/other_user}","gists_url":"https://api.github.com/users/Ttl/gists{/gist_id}","starred_url":"https://api.github.com/users/Ttl/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Ttl/subscriptions","organizations_url":"https://api.github.com/users/Ttl/orgs","repos_url":"https://api.github.com/users/Ttl/repos","events_url":"https://api.github.com/users/Ttl/events{/privacy}","received_events_url":"https://api.github.com/users/Ttl/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":5,"created_at":"2018-03-28T19:07:53Z","updated_at":"2018-03-28T20:45:52Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"It seems that the batch norm after the input convolution has very big variances. Minimum variance in the latest weights is 86 and maximum is 1.2e6. For comparison in LZ Go BN variances after input convolution are from 0.01 to 1.4. Much more sane values considering that BN will try to set the output variance to 1. If there are really that big values during training or inference it might cause issues with the floating point dynamic range.\r\n\r\nPlotting the summed absolute values of each 3x3 filter in input convolution gives following plot:\r\n \r\n![input_abs_sum_2d](https://user-images.githubusercontent.com/544240/38049620-60a84f46-32d1-11e8-8e59-8c422f8acea5.png)\r\n\r\nAnd sum across the output channels:\r\n\r\n![input_abs_sum](https://user-images.githubusercontent.com/544240/38049621-620c204c-32d1-11e8-92c9-11eb3ecb8a9f.png)\r\n\r\nMagnitude of weights for second to last input plane (move_count) are huge compared to everything else. Program is probably going to give completely different evaluations whether the commit https://github.com/glinscott/leela-chess/commit/fcd67bd5a4d34c4fceaa85ccf2264c33cf235ae0 is applied or not."},"comment":{"url":"https://api.github.com/repos/glinscott/leela-chess/issues/comments/377030315","html_url":"https://github.com/glinscott/leela-chess/issues/199#issuecomment-377030315","issue_url":"https://api.github.com/repos/glinscott/leela-chess/issues/199","id":377030315,"user":{"login":"Error323","id":76089,"avatar_url":"https://avatars0.githubusercontent.com/u/76089?v=4","gravatar_id":"","url":"https://api.github.com/users/Error323","html_url":"https://github.com/Error323","followers_url":"https://api.github.com/users/Error323/followers","following_url":"https://api.github.com/users/Error323/following{/other_user}","gists_url":"https://api.github.com/users/Error323/gists{/gist_id}","starred_url":"https://api.github.com/users/Error323/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Error323/subscriptions","organizations_url":"https://api.github.com/users/Error323/orgs","repos_url":"https://api.github.com/users/Error323/repos","events_url":"https://api.github.com/users/Error323/events{/privacy}","received_events_url":"https://api.github.com/users/Error323/received_events","type":"User","site_admin":false},"created_at":"2018-03-28T20:45:52Z","updated_at":"2018-03-28T20:45:52Z","author_association":"COLLABORATOR","body":"Could you revisualize the inputplanes @Ttl? Given this quick fix?"}}
{"actor":{"display_login":"Error323"}}