{"action":"created","issue":{"url":"https://api.github.com/repos/dmlc/mxnet/issues/4242","repository_url":"https://api.github.com/repos/dmlc/mxnet","labels_url":"https://api.github.com/repos/dmlc/mxnet/issues/4242/labels{/name}","comments_url":"https://api.github.com/repos/dmlc/mxnet/issues/4242/comments","events_url":"https://api.github.com/repos/dmlc/mxnet/issues/4242/events","html_url":"https://github.com/dmlc/mxnet/issues/4242","id":195725891,"number":4242,"title":"training process becomes zombie process and keeps occupying GPU memory","user":{"login":"nicklhy","id":1146226,"avatar_url":"https://avatars.githubusercontent.com/u/1146226?v=3","gravatar_id":"","url":"https://api.github.com/users/nicklhy","html_url":"https://github.com/nicklhy","followers_url":"https://api.github.com/users/nicklhy/followers","following_url":"https://api.github.com/users/nicklhy/following{/other_user}","gists_url":"https://api.github.com/users/nicklhy/gists{/gist_id}","starred_url":"https://api.github.com/users/nicklhy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nicklhy/subscriptions","organizations_url":"https://api.github.com/users/nicklhy/orgs","repos_url":"https://api.github.com/users/nicklhy/repos","events_url":"https://api.github.com/users/nicklhy/events{/privacy}","received_events_url":"https://api.github.com/users/nicklhy/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2016-12-15T06:19:16Z","updated_at":"2016-12-15T13:01:06Z","closed_at":null,"body":"My training program creates a child process for data IO while the model(mx.model.FeedForward) is running in the parent process. But for some unknown reason, I just find the parent process becomes a zombie process which keeps occupying the GPU memory.  The child process(for io) is still running background and can be killed.\r\n\r\nWhen I run `nvidia-smi` in cmd-line, part of the displayed message is \r\n+-----------------------------------------------------------------------------+\r\n| Processes:                                                       GPU Memory |\r\n|  GPU       PID  Type  Process name                               Usage      |\r\n|=====================================================================|\r\n|    0     31794    C   Unknown Error                                 5371MiB |\r\n|    1      7653    C   python                                        5369MiB |\r\n|    1     13386    C   /xx/xxx/caffe             1205MiB |\r\n|    1     27707    C   Unknown Error                                 1011MiB |\r\n|    2     18049    C   python                                        5809MiB |\r\n|    3     11827    C   python                                        6169MiB |\r\n+-----------------------------------------------------------------------------+\r\n\r\nIt seems that two different processes (31794 and 27707) became zombies at the same time (`top -p 31794/27707` shows the process's state is \"Z\").\r\n\r\nBTW, the parent of these two zombie processes is init(pid=1) now.\r\n\r\nDoes anyone seen this before? Is it possible to clear them and release the GPU memory without rebooting?\r\n\r\n------------\r\n## Environment info\r\nOperating System: Ubuntu 16.04 LTS\r\n\r\nCompiler: GCC 5.3\r\n\r\nPackage used (Python/R/Scala/Julia): Python\r\n\r\nOr if installed from source: Yes\r\n\r\nMXNet commit hash (`git rev-parse HEAD`): 2fefed4692007a34197df655f4e54db8942eb4ab\r\n\r\nPython version and distribution: Python 2.7"},"comment":{"url":"https://api.github.com/repos/dmlc/mxnet/issues/comments/267321974","html_url":"https://github.com/dmlc/mxnet/issues/4242#issuecomment-267321974","issue_url":"https://api.github.com/repos/dmlc/mxnet/issues/4242","id":267321974,"user":{"login":"inflation","id":2375962,"avatar_url":"https://avatars.githubusercontent.com/u/2375962?v=3","gravatar_id":"","url":"https://api.github.com/users/inflation","html_url":"https://github.com/inflation","followers_url":"https://api.github.com/users/inflation/followers","following_url":"https://api.github.com/users/inflation/following{/other_user}","gists_url":"https://api.github.com/users/inflation/gists{/gist_id}","starred_url":"https://api.github.com/users/inflation/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/inflation/subscriptions","organizations_url":"https://api.github.com/users/inflation/orgs","repos_url":"https://api.github.com/users/inflation/repos","events_url":"https://api.github.com/users/inflation/events{/privacy}","received_events_url":"https://api.github.com/users/inflation/received_events","type":"User","site_admin":false},"created_at":"2016-12-15T13:01:06Z","updated_at":"2016-12-15T13:01:06Z","body":"The system should take care of the process, as well as nvidia-smi. You can try to soft-reset the GPU. But it will also affect other processes. "}}
{"actor":{"display_login":"inflation"}}