{"action":"created","issue":{"url":"https://api.github.com/repos/mys007/ecc/issues/1","repository_url":"https://api.github.com/repos/mys007/ecc","labels_url":"https://api.github.com/repos/mys007/ecc/issues/1/labels{/name}","comments_url":"https://api.github.com/repos/mys007/ecc/issues/1/comments","events_url":"https://api.github.com/repos/mys007/ecc/issues/1/events","html_url":"https://github.com/mys007/ecc/issues/1","id":365324335,"node_id":"MDU6SXNzdWUzNjUzMjQzMzU=","number":1,"title":"Issues executing examples. CUDA_ERROR_ILLEGAL_ADDRESS and torch.bmm received an invalid combination of arguments","user":{"login":"dhorka","id":39189267,"node_id":"MDQ6VXNlcjM5MTg5MjY3","avatar_url":"https://avatars0.githubusercontent.com/u/39189267?v=4","gravatar_id":"","url":"https://api.github.com/users/dhorka","html_url":"https://github.com/dhorka","followers_url":"https://api.github.com/users/dhorka/followers","following_url":"https://api.github.com/users/dhorka/following{/other_user}","gists_url":"https://api.github.com/users/dhorka/gists{/gist_id}","starred_url":"https://api.github.com/users/dhorka/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dhorka/subscriptions","organizations_url":"https://api.github.com/users/dhorka/orgs","repos_url":"https://api.github.com/users/dhorka/repos","events_url":"https://api.github.com/users/dhorka/events{/privacy}","received_events_url":"https://api.github.com/users/dhorka/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2018-10-01T06:28:04Z","updated_at":"2018-10-24T08:07:59Z","closed_at":null,"author_association":"NONE","body":"Hi,\r\n\r\nI have some issues executing your code.  First, I tried to execute your example with modelnet 10 using the command provided. It seemed to work but an advanced epoch the code crash with this error:\r\n\r\n\r\n```\r\nTraceback (most recent call last):\r\n  File \"main.py\", line 315, in <module>\r\n    main()\r\n  File \"main.py\", line 217, in main\r\n    acc_train, loss, t_loader, t_trainer = train(epoch)\r\n  File \"main.py\", line 155, in train\r\n    loss_meter.add(loss.data[0])\r\nRuntimeError: cuda runtime error (77) : an illegal memory access was encountered at /pytorch/torch/lib/THC/generic/THCStorage.c:32\r\n\r\nTraceback (most recent call last):\r\n  File \"cupy/cuda/driver.pyx\", line 159, in cupy.cuda.driver.moduleUnload\r\n  File \"cupy/cuda/driver.pyx\", line 75, in cupy.cuda.driver.check_status\r\ncupy.cuda.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered\r\nException ignored in: 'cupy.cuda.function.Module.__dealloc__'\r\nTraceback (most recent call last):\r\n  File \"cupy/cuda/driver.pyx\", line 159, in cupy.cuda.driver.moduleUnload\r\n  File \"cupy/cuda/driver.pyx\", line 75, in cupy.cuda.driver.check_status\r\ncupy.cuda.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered\r\nTraceback (most recent call last):\r\n  File \"cupy/cuda/driver.pyx\", line 159, in cupy.cuda.driver.moduleUnload\r\n  File \"cupy/cuda/driver.pyx\", line 75, in cupy.cuda.driver.check_status\r\ncupy.cuda.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered\r\nException ignored in: 'cupy.cuda.function.Module.__dealloc__'\r\nTraceback (most recent call last):\r\n  File \"cupy/cuda/driver.pyx\", line 159, in cupy.cuda.driver.moduleUnload\r\n  File \"cupy/cuda/driver.pyx\", line 75, in cupy.cuda.driver.check_status\r\ncupy.cuda.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered\r\nTraceback (most recent call last):\r\n  File \"cupy/cuda/driver.pyx\", line 159, in cupy.cuda.driver.moduleUnload\r\n  File \"cupy/cuda/driver.pyx\", line 75, in cupy.cuda.driver.check_status\r\ncupy.cuda.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered\r\nException ignored in: 'cupy.cuda.function.Module.__dealloc__'\r\nTraceback (most recent call last):\r\n  File \"cupy/cuda/driver.pyx\", line 159, in cupy.cuda.driver.moduleUnload\r\n  File \"cupy/cuda/driver.pyx\", line 75, in cupy.cuda.driver.check_status\r\ncupy.cuda.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered\r\nTraceback (most recent call last):\r\n  File \"cupy/cuda/driver.pyx\", line 159, in cupy.cuda.driver.moduleUnload\r\n  File \"cupy/cuda/driver.pyx\", line 75, in cupy.cuda.driver.check_status\r\ncupy.cuda.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered\r\nException ignored in: 'cupy.cuda.function.Module.__dealloc__'\r\nTraceback (most recent call last):\r\n  File \"cupy/cuda/driver.pyx\", line 159, in cupy.cuda.driver.moduleUnload\r\n  File \"cupy/cuda/driver.pyx\", line 75, in cupy.cuda.driver.check_status\r\ncupy.cuda.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered\r\n=\r\n```\r\nI executed the code several times and the error appears randomly, it is not always in the same epoch, also it is not appearing in the same part of the code, here you can see an other example of the error:\r\n\r\n```\r\nFile \"main.py\", line 315, in <module>\r\n    main()\r\n  File \"main.py\", line 217, in main\r\n    acc_train, loss, t_loader, t_trainer = train(epoch)\r\n  File \"main.py\", line 152, in train\r\n    loss.backward()\r\n  File \"/projects/env/ecc/lib/python3.6/site-packages/torch/autograd/variable.py\", line 167, in backward\r\n    torch.autograd.backward(self, gradient, retain_graph, create_graph, retain_variables)\r\n  File \"/projects/env/ecc/lib/python3.6/site-packages/torch/autograd/__init__.py\", line 99, in backward\r\n    variables, grad_variables, retain_graph)\r\nRuntimeError: cublas runtime error : an internal operation failed at /pytorch/torch/lib/THC/THCBlas.cu:247\r\n\r\nTraceback (most recent call last):\r\n  File \"cupy/cuda/driver.pyx\", line 159, in cupy.cuda.driver.moduleUnload\r\n  File \"cupy/cuda/driver.pyx\", line 75, in cupy.cuda.driver.check_status\r\ncupy.cuda.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered\r\nException ignored in: 'cupy.cuda.function.Module.__dealloc__'\r\nTraceback (most recent call last):\r\n  File \"cupy/cuda/driver.pyx\", line 159, in cupy.cuda.driver.moduleUnload\r\n  File \"cupy/cuda/driver.pyx\", line 75, in cupy.cuda.driver.check_status\r\ncupy.cuda.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered\r\nTraceback (most recent call last):\r\n  File \"cupy/cuda/driver.pyx\", line 159, in cupy.cuda.driver.moduleUnload\r\n  File \"cupy/cuda/driver.pyx\", line 75, in cupy.cuda.driver.check_status\r\ncupy.cuda.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered\r\nException ignored in: 'cupy.cuda.function.Module.__dealloc__'\r\nTraceback (most recent call last):\r\n  File \"cupy/cuda/driver.pyx\", line 159, in cupy.cuda.driver.moduleUnload\r\n  File \"cupy/cuda/driver.pyx\", line 75, in cupy.cuda.driver.check_status\r\ncupy.cuda.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered\r\nTraceback (most recent call last):\r\n  File \"cupy/cuda/driver.pyx\", line 159, in cupy.cuda.driver.moduleUnload\r\n  File \"cupy/cuda/driver.pyx\", line 75, in cupy.cuda.driver.check_status\r\ncupy.cuda.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered\r\nException ignored in: 'cupy.cuda.function.Module.__dealloc__'\r\nTraceback (most recent call last):\r\n  File \"cupy/cuda/driver.pyx\", line 159, in cupy.cuda.driver.moduleUnload\r\n  File \"cupy/cuda/driver.pyx\", line 75, in cupy.cuda.driver.check_status\r\ncupy.cuda.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered\r\nTraceback (most recent call last):\r\n  File \"cupy/cuda/driver.pyx\", line 159, in cupy.cuda.driver.moduleUnload\r\n  File \"cupy/cuda/driver.pyx\", line 75, in cupy.cuda.driver.check_status\r\ncupy.cuda.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered\r\nException ignored in: 'cupy.cuda.function.Module.__dealloc__'\r\nTraceback (most recent call last):\r\n  File \"cupy/cuda/driver.pyx\", line 159, in cupy.cuda.driver.moduleUnload\r\n  File \"cupy/cuda/driver.pyx\", line 75, in cupy.cuda.driver.check_status\r\ncupy.cuda.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered\r\n```\r\n\r\n\r\nI tried different versions of pytorch: 0.2 0.3 0.4. The three versions was installed using pip, and also I tried to execute the code with a compiled from source version (0.2) the same error appears. I am using a machine with: 60gb of ram, Intel Xeon and a titan X with 12gb of ram. Moreover I tried to use different versions of open3d: 0.2.0 and 0.3.0. Finally I modified your sample command and I add edge_mem_limit in order to limit the memory used on the gpu without success.\r\n\r\nAlso I tested the code using the Sydney Urban Objects example, but in this case, this error is appearing at the begging of the execution:\r\n\r\n```\r\nFile \"main.py\", line 315, in <module>\r\n    main()\r\n  File \"main.py\", line 217, in main\r\n    acc_train, loss, t_loader, t_trainer = train(epoch)\r\n  File \"main.py\", line 148, in train\r\n    outputs = model(inputs)\r\n  File \"/project/env/ecc/lib/python3.6/site-packages/torch/nn/modules/module.py\", line 224, in __call__\r\n    result = self.forward(*input, **kwargs)\r\n  File \"/project/code/ecc/models.py\", line 103, in forward\r\n    input = module(input)\r\n  File \"/project/env/ecc/lib/python3.6/site-packages/torch/nn/modules/module.py\", line 224, in __call__\r\n    result = self.forward(*input, **kwargs)\r\n  File \"/project/code/ecc/ecc/GraphConvModule.py\", line 171, in forward\r\n    return GraphConvFunction(self._in_channels, self._out_channels, idxn, idxe, degs, degs_gpu, self._edge_mem_limit)(input, weights)\r\n  File \"/project/code/ecc/ecc/GraphConvModule.py\", line 63, in forward\r\n    self._multiply(sel_input, sel_weights, products, lambda a: a.unsqueeze(1))\r\n  File \"/project/code/ecc/ecc/GraphConvModule.py\", line 36, in _multiply\r\n    torch.bmm(f_a(a) if f_a else a, f_b(b) if f_b else b, out=out)\r\nTypeError: torch.bmm received an invalid combination of arguments - got (torch.DoubleTensor, torch.FloatTensor, out=torch.DoubleTensor), but expected (torch.DoubleTensor source, torch.DoubleTensor mat2, *, torch.DoubleTensor out)\r\n```\r\n\r\nPlease can you give me some hint in order to solve the issues?\r\n\r\nThanks,"},"comment":{"url":"https://api.github.com/repos/mys007/ecc/issues/comments/432555612","html_url":"https://github.com/mys007/ecc/issues/1#issuecomment-432555612","issue_url":"https://api.github.com/repos/mys007/ecc/issues/1","id":432555612,"node_id":"MDEyOklzc3VlQ29tbWVudDQzMjU1NTYxMg==","user":{"login":"dhorka","id":39189267,"node_id":"MDQ6VXNlcjM5MTg5MjY3","avatar_url":"https://avatars0.githubusercontent.com/u/39189267?v=4","gravatar_id":"","url":"https://api.github.com/users/dhorka","html_url":"https://github.com/dhorka","followers_url":"https://api.github.com/users/dhorka/followers","following_url":"https://api.github.com/users/dhorka/following{/other_user}","gists_url":"https://api.github.com/users/dhorka/gists{/gist_id}","starred_url":"https://api.github.com/users/dhorka/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dhorka/subscriptions","organizations_url":"https://api.github.com/users/dhorka/orgs","repos_url":"https://api.github.com/users/dhorka/repos","events_url":"https://api.github.com/users/dhorka/events{/privacy}","received_events_url":"https://api.github.com/users/dhorka/received_events","type":"User","site_admin":false},"created_at":"2018-10-24T08:07:59Z","updated_at":"2018-10-24T08:07:59Z","author_association":"NONE","body":"Hi,\r\n\r\n- Cupy version 4.3 pynvrtc 9.2\r\n- Cuda version: 8 cudnn: 6\r\n- I am running the command provided in the documentation for the modelnet10 this one:\r\n\r\n`python main.py \\\r\n--dataset modelnet10 --test_nth_epoch 25 --lr 0.1 --lr_steps '[50,100,150]' --epochs 175 --batch_size 64 --batch_parts 4 \\\r\n--model_config 'i_1_2, c_16,b,r, c_32,b,r, m_2.5_7.5, c_32,b,r, c_32,b,r, m_7.5_22.5, c_64,b,r, m_1e10_1e10, f_64,b,r,d_0.2,f_10' \\\r\n--fnet_llbias 0 --fnet_widths '[16,32]' --pc_augm_scale 1.2 --pc_augm_mirror_prob 0.2 --pc_augm_input_dropout 0.1 \\\r\n--nworkers 3 --edgecompaction 1 --odir results/modelnet10`"}}
{"actor":{"display_login":"dhorka"}}