{"action":"created","issue":{"url":"https://api.github.com/repos/pytorch/pytorch/issues/22671","repository_url":"https://api.github.com/repos/pytorch/pytorch","labels_url":"https://api.github.com/repos/pytorch/pytorch/issues/22671/labels{/name}","comments_url":"https://api.github.com/repos/pytorch/pytorch/issues/22671/comments","events_url":"https://api.github.com/repos/pytorch/pytorch/issues/22671/events","html_url":"https://github.com/pytorch/pytorch/issues/22671","id":466086165,"node_id":"MDU6SXNzdWU0NjYwODYxNjU=","number":22671,"title":"The training always freezes after some epochs.","user":{"login":"GeoffreyChen777","id":14183213,"node_id":"MDQ6VXNlcjE0MTgzMjEz","avatar_url":"https://avatars0.githubusercontent.com/u/14183213?v=4","gravatar_id":"","url":"https://api.github.com/users/GeoffreyChen777","html_url":"https://github.com/GeoffreyChen777","followers_url":"https://api.github.com/users/GeoffreyChen777/followers","following_url":"https://api.github.com/users/GeoffreyChen777/following{/other_user}","gists_url":"https://api.github.com/users/GeoffreyChen777/gists{/gist_id}","starred_url":"https://api.github.com/users/GeoffreyChen777/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/GeoffreyChen777/subscriptions","organizations_url":"https://api.github.com/users/GeoffreyChen777/orgs","repos_url":"https://api.github.com/users/GeoffreyChen777/repos","events_url":"https://api.github.com/users/GeoffreyChen777/events{/privacy}","received_events_url":"https://api.github.com/users/GeoffreyChen777/received_events","type":"User","site_admin":false},"labels":[{"id":1076930055,"node_id":"MDU6TGFiZWwxMDc2OTMwMDU1","url":"https://api.github.com/repos/pytorch/pytorch/labels/module:%20cuda","name":"module: cuda","color":"f7e101","default":false},{"id":553773019,"node_id":"MDU6TGFiZWw1NTM3NzMwMTk=","url":"https://api.github.com/repos/pytorch/pytorch/labels/needs%20reproduction","name":"needs reproduction","color":"cc9fe0","default":false},{"id":1303000678,"node_id":"MDU6TGFiZWwxMzAzMDAwNjc4","url":"https://api.github.com/repos/pytorch/pytorch/labels/topic:%20deadlock","name":"topic: deadlock","color":"0e8a16","default":false},{"id":1301347167,"node_id":"MDU6TGFiZWwxMzAxMzQ3MTY3","url":"https://api.github.com/repos/pytorch/pytorch/labels/triaged","name":"triaged","color":"006b75","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2019-07-10T04:00:14Z","updated_at":"2019-07-11T01:27:11Z","closed_at":null,"author_association":"NONE","body":"## 🐛 Bug\r\nThe training always freezes after some epochs. GPU usage is constantly 100%, the data loader also stops working. No error information.\r\n\r\nThe same problem is reported in #20611\r\n\r\n<!-- A clear and concise description of what the bug is. -->\r\n\r\n\r\n\r\n<!-- If you have a code sample, error messages, stack traces, please provide it here as well -->\r\n\r\n## Environment\r\n\r\n\r\nPyTorch version: 1.1.0                                                                                                                                                                         \r\nIs debug build: No                                                                                                                                                                             \r\nCUDA used to build PyTorch: 10.0.130                                                                                                                                                           \r\n                                                                                                                                                                                               \r\nOS: CentOS Linux 7 (Core)                                                                                                                                                                      \r\nGCC version: (GCC) 5.1.0                                                                                                                                                                       \r\nCMake version: Could not collect                                                                                                                                                               \r\n                                                                                                                                                                                               \r\nPython version: 3.6                                                                                                                                                                           \r\nIs CUDA available: Yes                                                                                                                                                                         \r\nCUDA runtime version: 10.0.130                                                                                                                                                                 \r\nGPU models and configuration:                                                                                                                                                                  \r\nGPU 0: Tesla V100-SXM2-16GB                                                                                                                                                                    \r\nGPU 1: Tesla V100-SXM2-16GB                                                                                                                                                                    \r\nGPU 2: Tesla V100-SXM2-16GB                                                                                                                                                                    \r\nGPU 3: Tesla V100-SXM2-16GB                                                                                                                                                                    \r\nGPU 4: Tesla V100-SXM2-16GB                                                                                                                                                                    \r\nGPU 5: Tesla V100-SXM2-16GB                                                                                                                                                                    \r\nGPU 6: Tesla V100-SXM2-16GB                                                                                                                                                                    \r\nGPU 7: Tesla V100-SXM2-16GB                                                                                                                                                                    \r\n                                                                                                                                                                                               \r\nNvidia driver version: 410.104                                                                                                                                                                 \r\ncuDNN version: /usr/local/cuda-10.0/targets/x86_64-linux/lib/libcudnn.so.7.4.1                                                                                                                 \r\n                                                                                                                                                                                               \r\nVersions of relevant libraries:                                                                                                                                                                \r\n[pip3] numpy==1.16.4                                                                                                                                                                           \r\n[pip3] torch==1.1.0                                                                                                                                                                            \r\n[pip3] torchvision==0.3.0                                                                                                                                                                      \r\n[conda] Could not collect\r\n"},"comment":{"url":"https://api.github.com/repos/pytorch/pytorch/issues/comments/510289705","html_url":"https://github.com/pytorch/pytorch/issues/22671#issuecomment-510289705","issue_url":"https://api.github.com/repos/pytorch/pytorch/issues/22671","id":510289705,"node_id":"MDEyOklzc3VlQ29tbWVudDUxMDI4OTcwNQ==","user":{"login":"GeoffreyChen777","id":14183213,"node_id":"MDQ6VXNlcjE0MTgzMjEz","avatar_url":"https://avatars0.githubusercontent.com/u/14183213?v=4","gravatar_id":"","url":"https://api.github.com/users/GeoffreyChen777","html_url":"https://github.com/GeoffreyChen777","followers_url":"https://api.github.com/users/GeoffreyChen777/followers","following_url":"https://api.github.com/users/GeoffreyChen777/following{/other_user}","gists_url":"https://api.github.com/users/GeoffreyChen777/gists{/gist_id}","starred_url":"https://api.github.com/users/GeoffreyChen777/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/GeoffreyChen777/subscriptions","organizations_url":"https://api.github.com/users/GeoffreyChen777/orgs","repos_url":"https://api.github.com/users/GeoffreyChen777/repos","events_url":"https://api.github.com/users/GeoffreyChen777/events{/privacy}","received_events_url":"https://api.github.com/users/GeoffreyChen777/received_events","type":"User","site_admin":false},"created_at":"2019-07-11T01:27:11Z","updated_at":"2019-07-11T01:27:11Z","author_association":"NONE","body":"This issue may be helpful. https://github.com/pytorch/fairseq/issues/708\r\n\r\nMy stack trace output is the same as https://github.com/pytorch/fairseq/issues/708 .\r\n\r\nThe fairseq repo have a try-cache block to handle the OOM problem.\r\n\r\nCan OOM cause this deadlock?"}}
{"actor":{"display_login":"GeoffreyChen777"}}