{"action":"closed","issue":{"url":"https://api.github.com/repos/pytorch/pytorch/issues/8283","repository_url":"https://api.github.com/repos/pytorch/pytorch","labels_url":"https://api.github.com/repos/pytorch/pytorch/issues/8283/labels{/name}","comments_url":"https://api.github.com/repos/pytorch/pytorch/issues/8283/comments","events_url":"https://api.github.com/repos/pytorch/pytorch/issues/8283/events","html_url":"https://github.com/pytorch/pytorch/issues/8283","id":330675648,"node_id":"MDU6SXNzdWUzMzA2NzU2NDg=","number":8283,"title":"Batchnorm gives different results depending on whether cudnn is enabled","user":{"login":"ssidorenko","id":1010462,"node_id":"MDQ6VXNlcjEwMTA0NjI=","avatar_url":"https://avatars2.githubusercontent.com/u/1010462?v=4","gravatar_id":"","url":"https://api.github.com/users/ssidorenko","html_url":"https://github.com/ssidorenko","followers_url":"https://api.github.com/users/ssidorenko/followers","following_url":"https://api.github.com/users/ssidorenko/following{/other_user}","gists_url":"https://api.github.com/users/ssidorenko/gists{/gist_id}","starred_url":"https://api.github.com/users/ssidorenko/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ssidorenko/subscriptions","organizations_url":"https://api.github.com/users/ssidorenko/orgs","repos_url":"https://api.github.com/users/ssidorenko/repos","events_url":"https://api.github.com/users/ssidorenko/events{/privacy}","received_events_url":"https://api.github.com/users/ssidorenko/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2018-06-08T14:15:11Z","updated_at":"2018-06-09T11:01:40Z","closed_at":"2018-06-09T11:01:40Z","author_association":"CONTRIBUTOR","body":"## Issue description\r\n\r\nI've run into issues when training a model of mine. It works fine when disabling cuDNN.\r\n\r\nI highly suspect the batchnorm layer to be the culprit. As seen in the code example below and the attached notebook, when cuDNN is enabled the results are inaccurate and non-deterministic.\r\n## Code example\r\n```\r\nimport torch\r\ntorch.backends.cudnn.enabled = False\r\n\r\nbn = torch.nn.BatchNorm1d(1).cuda()\r\nt = 100 * torch.ones([1,1,2]).cuda()\r\nprint(bn(t.cuda()))\r\n\r\n# prints tensor([[[ 0.,  0.]]], device='cuda:0')\r\n\r\ntorch.backends.cudnn.enabled = True\r\n\r\nbn = torch.nn.BatchNorm1d(1).cuda()\r\nt = 100 * torch.ones([1,1,2]).cuda()\r\nprint(bn(t.cuda()))\r\n\r\n# prints tensor(1.00000e-04 *       [[[ 8.5449,  8.5449]]], device='cuda:0')\r\n\r\n```\r\n## System Info\r\nPyTorch version: 0.4.0\r\nIs debug build: No\r\nCUDA used to build PyTorch: 9.1.85\r\n\r\nOS: Ubuntu 18.04 LTS\r\nGCC version: (Ubuntu 5.5.0-12ubuntu1) 5.5.0 20171010\r\nCMake version: version 3.9.4\r\n\r\nPython version: 3.6\r\nIs CUDA available: Yes\r\nCUDA runtime version: 9.1.85\r\nGPU models and configuration: GPU 0: GeForce GTX 1080 Ti\r\nNvidia driver version: 390.48\r\ncuDNN version: Probably one of the following:\r\n/usr/lib/x86_64-linux-gnu/libcudnn.so\r\n/usr/lib/x86_64-linux-gnu/libcudnn.so.7\r\n/usr/lib/x86_64-linux-gnu/libcudnn.so.7.1.2\r\n/usr/lib/x86_64-linux-gnu/libcudnn_static.a\r\n\r\nVersions of relevant libraries:\r\n[pip3] numpy (1.14.2)\r\n[pip3] numpydoc (0.7.0)\r\n[pip3] pytorchviz (0.0.1)\r\n[pip3] tensorboard-pytorch (0.7.1)\r\n[pip3] torch (0.4.0)\r\n[pip3] torchvision (0.2.0)\r\n[conda] cuda91                    1.0                  h4c16780_0    pytorch\r\n[conda] magma-cuda80              2.3.0                         1    pytorch\r\n[conda] pytorch                   0.4.0           py36_cuda9.1.85_cudnn7.1.2_1  [cuda91]  pytorch\r\n[conda] pytorchviz                0.0.1                     <pip>\r\n[conda] tensorboard-pytorch       0.7.1                     <pip>\r\n[conda] torchvision               0.2.1                    py36_1    pytorch\r\n"}}
{"actor":{"display_login":"apaszke"}}