{"action":"opened","issue":{"url":"https://api.github.com/repos/apache/incubator-mxnet/issues/10106","repository_url":"https://api.github.com/repos/apache/incubator-mxnet","labels_url":"https://api.github.com/repos/apache/incubator-mxnet/issues/10106/labels{/name}","comments_url":"https://api.github.com/repos/apache/incubator-mxnet/issues/10106/comments","events_url":"https://api.github.com/repos/apache/incubator-mxnet/issues/10106/events","html_url":"https://github.com/apache/incubator-mxnet/issues/10106","id":305142666,"number":10106,"title":"Get \"CUDA: an illegal memory access was encountered \"","user":{"login":"yancz1989","id":4943071,"avatar_url":"https://avatars0.githubusercontent.com/u/4943071?v=4","gravatar_id":"","url":"https://api.github.com/users/yancz1989","html_url":"https://github.com/yancz1989","followers_url":"https://api.github.com/users/yancz1989/followers","following_url":"https://api.github.com/users/yancz1989/following{/other_user}","gists_url":"https://api.github.com/users/yancz1989/gists{/gist_id}","starred_url":"https://api.github.com/users/yancz1989/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yancz1989/subscriptions","organizations_url":"https://api.github.com/users/yancz1989/orgs","repos_url":"https://api.github.com/users/yancz1989/repos","events_url":"https://api.github.com/users/yancz1989/events{/privacy}","received_events_url":"https://api.github.com/users/yancz1989/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-03-14T12:43:15Z","updated_at":"2018-03-14T12:43:15Z","closed_at":null,"author_association":"NONE","body":"Hi, all, I'm currently implementing a new operator. It runs smoothly using cpu, but under GPU, it get \r\nCheck failed: e == cudaSuccess CUDA: an illegal memory access was encountered\r\n\r\nAfter change MXNET_ENGINE_TYPE, the traceback shows\r\n\r\nTraceback (most recent call last):\r\n  File \"test.py\", line 199, in <module>\r\n    testTarget(conv1, int(sys.argv[1]))\r\n  File \"test.py\", line 167, in testTarget\r\n    output = mod.predict(val_iter)\r\n  File \"/usr/local/lib/python3.5/dist-packages/mxnet-1.1.0-py3.5.egg/mxnet/module/base_module.py\", line 354, in predict\r\n    outputs = [out[0:out.shape[0]-pad].copy() for out in self.get_outputs()]\r\n  File \"/usr/local/lib/python3.5/dist-packages/mxnet-1.1.0-py3.5.egg/mxnet/module/base_module.py\", line 354, in <listcomp>\r\n    outputs = [out[0:out.shape[0]-pad].copy() for out in self.get_outputs()]\r\n  File \"/usr/local/lib/python3.5/dist-packages/mxnet-1.1.0-py3.5.egg/mxnet/ndarray/ndarray.py\", line 1899, in copy\r\n    return self.copyto(self.context)\r\n  File \"/usr/local/lib/python3.5/dist-packages/mxnet-1.1.0-py3.5.egg/mxnet/ndarray/ndarray.py\", line 1879, in copyto\r\n    return _internal._copyto(self, out=hret)\r\n  File \"<string>\", line 25, in _copyto\r\n  File \"/usr/local/lib/python3.5/dist-packages/mxnet-1.1.0-py3.5.egg/mxnet/_ctypes/ndarray.py\", line 92, in _imperative_invoke\r\n    ctypes.byref(out_stypes)))\r\n  File \"/usr/local/lib/python3.5/dist-packages/mxnet-1.1.0-py3.5.egg/mxnet/base.py\", line 146, in check_call\r\n    raise MXNetError(py_str(_LIB.MXGetLastError()))\r\nmxnet.base.MXNetError: [20:12:22] /mnt/disk1/xxx/rdr/dev/mxnet/mshadow/mshadow/./stream_gpu-inl.h:62: Check failed: e == cudaSuccess CUDA: an illegal memory access was encountered\r\n\r\nI log one \"complete\" after each cuda parallel kernel, and it seems it finish all the computing, but the get_outputs runs a failure.\r\n\r\nCan someone give a hint on how such thing happens? Thanks for any advice."}}
{"actor":{"display_login":"yancz1989"}}