{"action":"created","issue":{"url":"https://api.github.com/repos/pytorch/pytorch/issues/13276","repository_url":"https://api.github.com/repos/pytorch/pytorch","labels_url":"https://api.github.com/repos/pytorch/pytorch/issues/13276/labels{/name}","comments_url":"https://api.github.com/repos/pytorch/pytorch/issues/13276/comments","events_url":"https://api.github.com/repos/pytorch/pytorch/issues/13276/events","html_url":"https://github.com/pytorch/pytorch/issues/13276","id":375246905,"node_id":"MDU6SXNzdWUzNzUyNDY5MDU=","number":13276,"title":"Memory error for batched inverse","user":{"login":"MarcoForte","id":540338,"node_id":"MDQ6VXNlcjU0MDMzOA==","avatar_url":"https://avatars1.githubusercontent.com/u/540338?v=4","gravatar_id":"","url":"https://api.github.com/users/MarcoForte","html_url":"https://github.com/MarcoForte","followers_url":"https://api.github.com/users/MarcoForte/followers","following_url":"https://api.github.com/users/MarcoForte/following{/other_user}","gists_url":"https://api.github.com/users/MarcoForte/gists{/gist_id}","starred_url":"https://api.github.com/users/MarcoForte/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/MarcoForte/subscriptions","organizations_url":"https://api.github.com/users/MarcoForte/orgs","repos_url":"https://api.github.com/users/MarcoForte/repos","events_url":"https://api.github.com/users/MarcoForte/events{/privacy}","received_events_url":"https://api.github.com/users/MarcoForte/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":6,"created_at":"2018-10-29T22:41:49Z","updated_at":"2018-10-30T19:52:37Z","closed_at":null,"author_association":"NONE","body":"## 🐛 Bug\r\n\r\nThere seems to be a limit of N=256x256 for the number of batches one can invert on the gpu. I don't think this limit corresponds properly to available memory. Because I can run a N=256x256 -1 batched inverse on a gpu with less availiable memory. I have tested tensorflow on google collab and it allows batched inverse of size N=6144x6144.\r\n\r\n## To Reproduce\r\n\r\n```\r\nimport torch\r\nN = 256\r\nx = torch.randn(N*N , 2, 2).cuda()\r\ny = torch.inverse(x)\r\nprint(y.cpu().numpy())\r\n```\r\n\r\nError:\r\n\r\n> ----> 1 y.cpu().numpy()\r\n> \r\n> RuntimeError: cuda runtime error (9) : invalid configuration argument at /opt/conda/conda-bld/pytorch-nightly_1540802486426/work/aten/src/THC/THCTensorCopy.cu:205\r\n\r\n## Expected behavior\r\nIt works if I reduce the batch dimension by one.\r\n`x = torch.randn(N*N -1 , 2, 2).cuda()\r\n`\r\n## Environment\r\n\r\n> PyTorch version: 1.0.0.dev20181029\r\n> Is debug build: No\r\n> CUDA used to build PyTorch: 8.0.61\r\n> \r\n> OS: Ubuntu 16.04.4 LTS\r\n> GCC version: (Ubuntu 5.4.0-6ubuntu1~16.04.2) 5.4.0 20160609\r\n> CMake version: version 3.11.0\r\n> \r\n> Python version: 3.6\r\n> Is CUDA available: Yes\r\n> CUDA runtime version: 9.1.85\r\n> GPU models and configuration: \r\n> GPU 0: GeForce GTX 1080\r\n> GPU 1: GeForce GTX 1080\r\n> \r\n> Nvidia driver version: 387.26\r\n> cuDNN version: Probably one of the following:\r\n> /usr/local/MATLAB/R2016b/bin/glnxa64/libcudnn.so.4.0.7\r\n> /usr/local/cuda-9.1/lib64/libcudnn.so\r\n> /usr/local/cuda-9.1/lib64/libcudnn.so.7\r\n> /usr/local/cuda-9.1/lib64/libcudnn.so.7.0.5\r\n> /usr/local/cuda-9.1/lib64/libcudnn.so.7.1.1\r\n> /usr/local/cuda-9.1/lib64/libcudnn_static.a\r\n> \r\n> Versions of relevant libraries:\r\n> [pip] Could not collect\r\n> [conda] cuda80                    1.0                           0    soumith\r\n> [conda] cuda91                    1.0                  h4c16780_0    pytorch\r\n> [conda] guided-filter-pytorch     1.1.1                     <pip>\r\n> [conda] magma-cuda91              2.3.0                         1    pytorch\r\n> [conda] pytorch-nightly           1.0.0.dev20181029 py3.6_cuda8.0.61_cudnn7.1.2_0  [cuda80]  pytorch\r\n> [conda] torch                     0.4.0                     <pip>\r\n> [conda] torch                     0.3.1                     <pip>\r\n"},"comment":{"url":"https://api.github.com/repos/pytorch/pytorch/issues/comments/434445241","html_url":"https://github.com/pytorch/pytorch/issues/13276#issuecomment-434445241","issue_url":"https://api.github.com/repos/pytorch/pytorch/issues/13276","id":434445241,"node_id":"MDEyOklzc3VlQ29tbWVudDQzNDQ0NTI0MQ==","user":{"login":"zou3519","id":5652049,"node_id":"MDQ6VXNlcjU2NTIwNDk=","avatar_url":"https://avatars3.githubusercontent.com/u/5652049?v=4","gravatar_id":"","url":"https://api.github.com/users/zou3519","html_url":"https://github.com/zou3519","followers_url":"https://api.github.com/users/zou3519/followers","following_url":"https://api.github.com/users/zou3519/following{/other_user}","gists_url":"https://api.github.com/users/zou3519/gists{/gist_id}","starred_url":"https://api.github.com/users/zou3519/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zou3519/subscriptions","organizations_url":"https://api.github.com/users/zou3519/orgs","repos_url":"https://api.github.com/users/zou3519/repos","events_url":"https://api.github.com/users/zou3519/events{/privacy}","received_events_url":"https://api.github.com/users/zou3519/received_events","type":"User","site_admin":false},"created_at":"2018-10-30T19:52:37Z","updated_at":"2018-10-30T19:52:37Z","author_association":"CONTRIBUTOR","body":"I have another independent report that this can be reproduced using the magma build that is installed alongside pytorch (magma-cudaXX) where XX is the cuda version. It could be a bug in the magma version we ship with"}}
{"actor":{"display_login":"zou3519"}}