{"action":"created","issue":{"url":"https://api.github.com/repos/pytorch/pytorch/issues/3022","repository_url":"https://api.github.com/repos/pytorch/pytorch","labels_url":"https://api.github.com/repos/pytorch/pytorch/issues/3022/labels{/name}","comments_url":"https://api.github.com/repos/pytorch/pytorch/issues/3022/comments","events_url":"https://api.github.com/repos/pytorch/pytorch/issues/3022/events","html_url":"https://github.com/pytorch/pytorch/issues/3022","id":263698513,"number":3022,"title":"Reliably repeating pytorch system crash/reboot when using imagenet examples","user":{"login":"castleguarders","id":25285331,"avatar_url":"https://avatars3.githubusercontent.com/u/25285331?v=4","gravatar_id":"","url":"https://api.github.com/users/castleguarders","html_url":"https://github.com/castleguarders","followers_url":"https://api.github.com/users/castleguarders/followers","following_url":"https://api.github.com/users/castleguarders/following{/other_user}","gists_url":"https://api.github.com/users/castleguarders/gists{/gist_id}","starred_url":"https://api.github.com/users/castleguarders/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/castleguarders/subscriptions","organizations_url":"https://api.github.com/users/castleguarders/orgs","repos_url":"https://api.github.com/users/castleguarders/repos","events_url":"https://api.github.com/users/castleguarders/events{/privacy}","received_events_url":"https://api.github.com/users/castleguarders/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":11,"created_at":"2017-10-08T07:06:33Z","updated_at":"2018-04-19T16:49:30Z","closed_at":"2017-10-10T13:49:03Z","author_association":"NONE","body":"So I have a 100% repeatable system crash (reboot) when trying to run the imagenet example (2012 dataset). resnet18 defaults. The crash seems to happen at Variable.py at torch.autograd.backward(..) (line 158).\r\n\r\nI am able to run the basic mnist example successfully. \r\n\r\n Setup: Ubuntu 16.04, 4.10.0-35-generic #39~16.04.1-Ubuntu SMP Wed Sep 13 09:02:42 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux\r\n\r\npython --version Python 3.6.2 :: Anaconda, Inc.\r\n\r\n/usr/local/cuda/bin/nvcc --version\r\nnvcc: NVIDIA (R) Cuda compiler driver\r\nCopyright (c) 2005-2017 NVIDIA Corporation\r\nBuilt on Fri_Sep__1_21:08:03_CDT_2017\r\nCuda compilation tools, release 9.0, V9.0.176\r\n\r\nnvidia-smi output.\r\nSat Oct  7 23:51:53 2017       \r\n+-----------------------------------------------------------------------------+\r\n| NVIDIA-SMI 384.81                 Driver Version: 384.81                    |\r\n|-------------------------------+----------------------+----------------------+\r\n| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\r\n| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\r\n|===============================+======================+======================|\r\n|   0  GeForce GTX 108...  Off  | 00000000:03:00.0  On |                  N/A |\r\n| 14%   51C    P8    18W / 250W |    650MiB / 11170MiB |      0%      Default |\r\n+-------------------------------+----------------------+----------------------+\r\n                                                                               \r\n+-----------------------------------------------------------------------------+\r\n| Processes:                                                       GPU Memory |\r\n|  GPU       PID   Type   Process name                             Usage      |\r\n|=============================================================================|\r\n|    0      1335      G   /usr/lib/xorg/Xorg                           499MiB |\r\n|    0      2231      G   cinnamon                                      55MiB |\r\n|    0      3390      G   ...-token=C6DE372B6D9D4FCD6453869AF4C6B4E5    93MiB |\r\n+-----------------------------------------------------------------------------+\r\n\r\ntorch/vision was built locally on the machine from master. No issues at compile or install time, other than the normal compile time warnings...\r\n\r\nHappy to help get further information..\r\n"},"comment":{"url":"https://api.github.com/repos/pytorch/pytorch/issues/comments/382805095","html_url":"https://github.com/pytorch/pytorch/issues/3022#issuecomment-382805095","issue_url":"https://api.github.com/repos/pytorch/pytorch/issues/3022","id":382805095,"user":{"login":"pmcrodrigues","id":25279086,"avatar_url":"https://avatars3.githubusercontent.com/u/25279086?v=4","gravatar_id":"","url":"https://api.github.com/users/pmcrodrigues","html_url":"https://github.com/pmcrodrigues","followers_url":"https://api.github.com/users/pmcrodrigues/followers","following_url":"https://api.github.com/users/pmcrodrigues/following{/other_user}","gists_url":"https://api.github.com/users/pmcrodrigues/gists{/gist_id}","starred_url":"https://api.github.com/users/pmcrodrigues/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pmcrodrigues/subscriptions","organizations_url":"https://api.github.com/users/pmcrodrigues/orgs","repos_url":"https://api.github.com/users/pmcrodrigues/repos","events_url":"https://api.github.com/users/pmcrodrigues/events{/privacy}","received_events_url":"https://api.github.com/users/pmcrodrigues/received_events","type":"User","site_admin":false},"created_at":"2018-04-19T16:49:30Z","updated_at":"2018-04-19T16:49:30Z","author_association":"NONE","body":"@castleguarders @yurymalkov Thank you both. I have also tried to reduce power draw via nvidia-smi and it stopped crashing the system. But with stress tests at full power draw simultaneously on my 2 xeons (with http://people.seas.harvard.edu/~apw/stress/) and the 4 1080ti (with https://github.com/wilicc/gpu-burn) didn't make it crash. So for now I have only seen this problem on pytorch. Maybe I need other stress tests?"}}
{"actor":{"display_login":"pmcrodrigues"}}