{"action":"created","issue":{"url":"https://api.github.com/repos/NVIDIA/apex/issues/43","repository_url":"https://api.github.com/repos/NVIDIA/apex","labels_url":"https://api.github.com/repos/NVIDIA/apex/issues/43/labels{/name}","comments_url":"https://api.github.com/repos/NVIDIA/apex/issues/43/comments","events_url":"https://api.github.com/repos/NVIDIA/apex/issues/43/events","html_url":"https://github.com/NVIDIA/apex/issues/43","id":360385466,"node_id":"MDU6SXNzdWUzNjAzODU0NjY=","number":43,"title":"resnet50 doesn't converge when running example/imagenet/main.py on imagenet dataset with fp16","user":{"login":"wangdongxuking61","id":9778128,"node_id":"MDQ6VXNlcjk3NzgxMjg=","avatar_url":"https://avatars1.githubusercontent.com/u/9778128?v=4","gravatar_id":"","url":"https://api.github.com/users/wangdongxuking61","html_url":"https://github.com/wangdongxuking61","followers_url":"https://api.github.com/users/wangdongxuking61/followers","following_url":"https://api.github.com/users/wangdongxuking61/following{/other_user}","gists_url":"https://api.github.com/users/wangdongxuking61/gists{/gist_id}","starred_url":"https://api.github.com/users/wangdongxuking61/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wangdongxuking61/subscriptions","organizations_url":"https://api.github.com/users/wangdongxuking61/orgs","repos_url":"https://api.github.com/users/wangdongxuking61/repos","events_url":"https://api.github.com/users/wangdongxuking61/events{/privacy}","received_events_url":"https://api.github.com/users/wangdongxuking61/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2018-09-14T17:10:18Z","updated_at":"2018-09-17T14:17:06Z","closed_at":null,"author_association":"NONE","body":"i want use `example/imagenet/main.py` to train resnet50 model on imagenet dataset with fp16. But the accurary can't  converge. BTW, not using fp16 will get right top1 accurary=76%.\r\n\r\nmy command is:\r\n\r\n```\r\npython -m torch.distributed.launch --nproc_per_node=8 main.py --fp16 --arch resnet50 --epochs 90 --workers 6 --batch-size=256 /imagenet\r\n```\r\n\r\n- Python version: 3.6.2\r\n\r\n- PyTorch version: 0.4.1\r\n\r\n- torchvision version: 0.2.1\r\n\r\n  \r\n\r\n- OS: Ubuntu 16.04.3 LTS\r\n\r\n- Nvidia driver version: 390.46\r\n\r\n- CUDA runtime version: 9.0\r\n\r\n- GPU number: 8\r\n\r\n- GPU model: Tesla P100-PCIE\r\n\r\n\r\n\r\nthe validate accurary suddenly fall down to 0 after about 7 epochs.\r\nthe train accurary suddenly fall down to 0 after about 17 epochs.\r\n\r\ni saved model's gradient( `para.grad` )each epoch,  i found when epoch=17, the data distribution of model's parameter( `para.data`） is normal, but 84.5% of gradient data( `para.grad`) is NaN.\r\n\r\n\r\n\r\nthe accurary result as follows,\r\n\r\n| train |        |        |        |      | validate |        |        |          |\r\n| ----- | ------ | ------ | ------ | ---- | -------- | ------ | ------ | -------- |\r\n| epoch | Top1   | Top5   | Loss   |      | epoch    | Top1   | Top5   | Loss     |\r\n| 0     | 3.166  | 9.401  | 6.0748 |      | 0        | 3.054  | 9.308  | 12.1711  |\r\n| 1     | 15.428 | 34.406 | 4.4438 |      | 1        | 18.206 | 39.778 | 4.1692   |\r\n| 2     | 26.628 | 50.356 | 3.6012 |      | 2        | 29.108 | 54.764 | 3.3741   |\r\n| 3     | 34.069 | 59.227 | 3.1205 |      | 3        | 30.938 | 56.796 | 3.2538   |\r\n| 4     | 37.787 | 63.202 | 2.8991 |      | 4        | 29.46  | 55.652 | 3.3615   |\r\n| 5     | 40.33  | 65.834 | 2.7536 |      | 5        | 12.982 | 30.574 | 5.1418   |\r\n| 6     | 42.476 | 67.836 | 2.6325 |      | 6        | 0.428  | 1.608  | 8.4904   |\r\n| 7     | 43.851 | 69.086 | 2.5574 |      | 7        | 0.1    | 0.502  | 8.2962   |\r\n| 8     | 44.888 | 70.058 | 2.5005 |      | 8        | 0.1    | 0.49   | 15.8809  |\r\n| 9     | 45.692 | 70.684 | 2.4588 |      | 9        | 0.1    | 0.5    | 83.5319  |\r\n| 10    | 46.378 | 71.274 | 2.4261 |      | 10       | 0.104  | 0.496  | 184.0083 |\r\n| 11    | 46.66  | 71.618 | 2.4065 |      | 11       | 0.1    | 0.504  | 210.9373 |\r\n| 12    | 46.938 | 71.805 | 2.3928 |      | 12       | 0.1    | 0.5    | 585.1285 |\r\n| 13    | 47.039 | 71.931 | 2.3873 |      | 13       | 0.1    | 0.5    | 2283.96  |\r\n| 14    | 46.974 | 71.87  | 2.393  |      | 14       | 0      | 0.006  | 1612.295 |\r\n| 15    | 46.667 | 71.499 | 2.4104 |      | 15       | 0.002  | 0.006  | 7.0508   |\r\n| 16    | 46.273 | 71.155 | 2.4337 |      | 16       | 0.002  | 0.006  | 1554.635 |\r\n| 17    | 16.3   | 25.251 | 5.3414 |      | 17       | 0.1    | 0.5    | 8.9353   |\r\n| 18    | 0.096  | 0.482  | 6.9067 |      | 18       | 0.1    | 0.5    | 7.0235   |\r\n| 19    | 0.095  | 0.485  | 6.9068 |      | 19       | 0.1    | 0.5    | 6.911    |\r\n| 20    | 0.097  | 0.488  | 6.9068 |      | 20       | 0.1    | 0.5    | 6.9091   |\r\n| 21    | 0.094  | 0.491  | 6.9067 |      | 21       | 0.1    | 0.5    | 6.9086   |\r\n| 22    | 0.094  | 0.487  | 6.9066 |      | 22       | 0.1    | 0.5    | 6.9085   |\r\n| 23    | 0.095  | 0.478  | 6.9066 |      | 23       | 0.1    | 0.5    | 6.9085   |\r\n| 24    | 0.101  | 0.491  | 6.9067 |      | 24       | 0.1    | 0.5    | 6.9082   |\r\n| 25    | 0.098  | 0.487  | 6.9067 |      | 25       | 0.1    | 0.5    | 6.9083   |\r\n| 26    | 0.097  | 0.483  | 6.9068 |      | 26       | 0.1    | 0.5    | 6.908    |\r\n| 27    | 0.099  | 0.485  | 6.9067 |      | 27       | 0.1    | 0.5    | 6.9082   |\r\n| 28    | 0.091  | 0.489  | 6.9067 |      | 28       | 0.1    | 0.5    | 6.9085   |\r\n| 29    | 0.097  | 0.489  | 6.9067 |      | 29       | 0.1    | 0.5    | 6.9083   |\r\n| 30    | 0.1    | 0.503  | 6.9065 |      | 30       | 0.1    | 0.5    | 6.908    |\r\n| 31    | 0.1    | 0.496  | 6.9063 |      | 31       | 0.1    | 0.5    | 6.9078   |\r\n| 32    | 0.098  | 0.487  | 6.9063 |      | 32       | 0.1    | 0.5    | 6.908    |\r\n| 33    | 0.092  | 0.472  | 6.9063 |      | 33       | 0.1    | 0.5    | 6.9078   |\r\n| 34    | 0.092  | 0.469  | 6.9063 |      | 34       | 0.1    | 0.5    | 6.9078   |\r\n| 35    | 0.095  | 0.461  | 6.9063 |      | 35       | 0.1    | 0.5    | 6.9078   |\r\n| 36    | 0.093  | 0.463  | 6.9063 |      | 36       | 0.1    | 0.5    | 6.9078   |\r\n| 37    | 0.086  | 0.459  | 6.9062 |      | 37       | 0.1    | 0.5    | 6.908    |\r\n| 38    | 0.089  | 0.467  | 6.9063 |      | 38       | 0.1    | 0.5    | 6.9078   |\r\n| 39    | 0.092  | 0.469  | 6.9063 |      | 39       | 0.1    | 0.5    | 6.908    |\r\n| 40    | 0.092  | 0.461  | 6.9063 |      | 40       | 0.1    | 0.5    | 6.908    |\r\n| 41    | 0.095  | 0.459  | 6.9063 |      | 41       | 0.1    | 0.5    | 6.9078   |\r\n| 42    | 0.09   | 0.46   | 6.9063 |      | 42       | 0.1    | 0.5    | 6.9078   |\r\n| 43    | 0.09   | 0.461  | 6.9063 |      | 43       | 0.1    | 0.5    | 6.908    |\r\n| 44    | 0.093  | 0.463  | 6.9063 |      | 44       | 0.1    | 0.5    | 6.908    |\r\n| 45    | 0.094  | 0.464  | 6.9063 |      | 45       | 0.1    | 0.5    | 6.9078   |\r\n| 46    | 0.09   | 0.457  | 6.9063 |      | 46       | 0.1    | 0.5    | 6.9078   |\r\n| 47    | 0.091  | 0.466  | 6.9063 |      | 47       | 0.1    | 0.5    | 6.908    |\r\n| 48    | 0.089  | 0.465  | 6.9063 |      | 48       | 0.1    | 0.5    | 6.9078   |\r\n| 49    | 0.09   | 0.449  | 6.9063 |      | 49       | 0.1    | 0.5    | 6.9078   |\r\n| 50    | 0.094  | 0.46   | 6.9063 |      | 50       | 0.1    | 0.5    | 6.9078   |\r\n| 51    | 0.094  | 0.464  | 6.9063 |      | 51       | 0.1    | 0.5    | 6.908    |\r\n| 52    | 0.092  | 0.473  | 6.9063 |      | 52       | 0.1    | 0.5    | 6.9078   |\r\n| 53    | 0.094  | 0.462  | 6.9063 |      | 53       | 0.1    | 0.5    | 6.9078   |\r\n| 54    | 0.088  | 0.468  | 6.9063 |      | 54       | 0.1    | 0.5    | 6.9078   |\r\n| 55    | 0.091  | 0.453  | 6.9063 |      | 55       | 0.1    | 0.5    | 6.9078   |\r\n| 56    | 0.091  | 0.45   | 6.9064 |      | 56       | 0.1    | 0.5    | 6.9078   |\r\n| 57    | 0.093  | 0.472  | 6.9063 |      | 57       | 0.1    | 0.5    | 6.9077   |\r\n| 58    | 0.09   | 0.455  | 6.9063 |      | 58       | 0.1    | 0.5    | 6.9077   |\r\n| 59    | 0.091  | 0.464  | 6.9063 |      | 59       | 0.1    | 0.5    | 6.9078   |\r\n| 60    | 0.099  | 0.491  | 6.9063 |      | 60       | 0.1    | 0.5    | 6.9078   |\r\n| 61    | 0.101  | 0.499  | 6.9063 |      | 61       | 0.1    | 0.5    | 6.908    |\r\n| 62    | 0.1    | 0.496  | 6.9063 |      | 62       | 0.1    | 0.5    | 6.908    |\r\n| 63    | 0.1    | 0.487  | 6.9063 |      | 63       | 0.1    | 0.5    | 6.9082   |\r\n| 64    | 0.098  | 0.483  | 6.9063 |      | 64       | 0.1    | 0.5    | 6.9082   |\r\n| 65    | 0.094  | 0.461  | 6.9064 |      | 65       | 0.1    | 0.5    | 6.9082   |\r\n| 66    | 0.091  | 0.467  | 6.9063 |      | 66       | 0.1    | 0.5    | 6.9082   |\r\n| 67    | 0.093  | 0.466  | 6.9064 |      | 67       | 0.1    | 0.5    | 6.9082   |\r\n| 68    | 0.097  | 0.471  | 6.9063 |      | 68       | 0.1    | 0.5    | 6.9082   |\r\n| 69    | 0.088  | 0.461  | 6.9064 |      | 69       | 0.1    | 0.5    | 6.9082   |\r\n| 70    | 0.093  | 0.459  | 6.9063 |      | 70       | 0.1    | 0.5    | 6.9082   |\r\n| 71    | 0.096  | 0.473  | 6.9064 |      | 71       | 0.1    | 0.5    | 6.9083   |\r\n| 72    | 0.092  | 0.471  | 6.9064 |      | 72       | 0.1    | 0.5    | 6.9082   |\r\n| 73    | 0.095  | 0.464  | 6.9064 |      | 73       | 0.1    | 0.5    | 6.9083   |\r\n| 74    | 0.092  | 0.464  | 6.9063 |      | 74       | 0.1    | 0.5    | 6.9083   |\r\n| 75    | 0.09   | 0.462  | 6.9064 |      | 75       | 0.1    | 0.5    | 6.9083   |\r\n| 76    | 0.093  | 0.467  | 6.9064 |      | 76       | 0.1    | 0.5    | 6.9083   |\r\n| 77    | 0.091  | 0.467  | 6.9064 |      | 77       | 0.1    | 0.5    | 6.9083   |\r\n| 78    | 0.092  | 0.455  | 6.9064 |      | 78       | 0.1    | 0.5    | 6.9083   |\r\n| 79    | 0.09   | 0.459  | 6.9064 |      | 79       | 0.1    | 0.5    | 6.9082   |\r\n| 80    | 0.095  | 0.493  | 6.9064 |      | 80       | 0.1    | 0.5    | 6.9082   |\r\n| 81    | 0.094  | 0.486  | 6.9064 |      | 81       | 0.1    | 0.5    | 6.9082   |\r\n| 82    | 0.099  | 0.487  | 6.9064 |      | 82       | 0.1    | 0.5    | 6.9082   |\r\n| 83    | 0.094  | 0.498  | 6.9064 |      | 83       | 0.1    | 0.5    | 6.9082   |\r\n| 84    | 0.096  | 0.492  | 6.9064 |      | 84       | 0.1    | 0.5    | 6.9082   |\r\n| 85    | 0.097  | 0.487  | 6.9064 |      | 85       | 0.1    | 0.5    | 6.9083   |\r\n| 86    | 0.096  | 0.492  | 6.9064 |      | 86       | 0.1    | 0.5    | 6.9082   |\r\n| 87    | 0.1    | 0.493  | 6.9065 |      | 87       | 0.1    | 0.5    | 6.9083   |\r\n| 88    | 0.099  | 0.482  | 6.9064 |      | 88       | 0.1    | 0.5    | 6.9083   |\r\n| 89    | 0.097  | 0.498  | 6.9064 |      | 89       | 0.1    | 0.5    | 6.9082   |\r\n\r\nwhen i got the wrong result, i run the main.py on another server with two V100. I only use one, because when i want use two, i got the trouble described in [pytorch issue 11327](https://github.com/pytorch/pytorch/issues/11327#). But it does not affect my training...\r\n\r\n```\r\npython main.py --fp16 --arch resnet50 --epochs 90 --workers 6 --batch-size=256 /imagenet\r\n```\r\n\r\nThen i got the same wrong result, it is very like the above results. \r\n\r\n\r\n\r\nI want to know whether you have tested the `main.py`on imagenet dataset with `fp16`,and got a good accurary, like top1=76% described in the paper [MIXED PRECISION TRAINING](https://arxiv.org/pdf/1710.03740.pdf)."},"comment":{"url":"https://api.github.com/repos/NVIDIA/apex/issues/comments/422034866","html_url":"https://github.com/NVIDIA/apex/issues/43#issuecomment-422034866","issue_url":"https://api.github.com/repos/NVIDIA/apex/issues/43","id":422034866,"node_id":"MDEyOklzc3VlQ29tbWVudDQyMjAzNDg2Ng==","user":{"login":"mcarilli","id":7799218,"node_id":"MDQ6VXNlcjc3OTkyMTg=","avatar_url":"https://avatars0.githubusercontent.com/u/7799218?v=4","gravatar_id":"","url":"https://api.github.com/users/mcarilli","html_url":"https://github.com/mcarilli","followers_url":"https://api.github.com/users/mcarilli/followers","following_url":"https://api.github.com/users/mcarilli/following{/other_user}","gists_url":"https://api.github.com/users/mcarilli/gists{/gist_id}","starred_url":"https://api.github.com/users/mcarilli/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mcarilli/subscriptions","organizations_url":"https://api.github.com/users/mcarilli/orgs","repos_url":"https://api.github.com/users/mcarilli/repos","events_url":"https://api.github.com/users/mcarilli/events{/privacy}","received_events_url":"https://api.github.com/users/mcarilli/received_events","type":"User","site_admin":false},"created_at":"2018-09-17T14:17:06Z","updated_at":"2018-09-17T14:17:06Z","author_association":"CONTRIBUTOR","body":"Glad to hear it!  \r\n\r\nHonestly, i don’t know why a static loss scale of 1 didn’t work.  The run in the paper may have used different versions of cudnn or cublas with different stability properties.  In general, loss scaling is usually helpful, so there’s no reason not to use it, especially when Apex tools can handle it for you automatically."}}
{"actor":{"display_login":"mcarilli"}}