title: CUDA crashes under some certain situation
body: <p>I am using pytorch 1.8.2 LTS + CUDA 11.1 + CuDNN 8.0.4 under Ubuntu 18.04. It crashes oddly under some certain situation. The problem can be reproduced as following.</p>
  <pre><code>import torch
  device = &quot;cuda&quot; if torch.cuda.is_available() else &quot;cpu&quot;
  a = torch.randn(2, 4).to(device)
  b = torch.randn(4, 2).to(device)
  c = torch.matmul(a, b)
  a = torch.randn(100, 64).to(device)
  b = torch.randn(64, 100).to(device)
  c = torch.matmul(a, b)
  a = torch.randn(1, 100, 64).to(device)
  b = torch.randn(1, 64, 100).to(device)
  torch.matmul(a, b)
  
  tensor([[[  8.1834,   1.8383,   1.7945,  ...,  12.6254,  12.8753, -11.5631],
           [  3.8268,  -7.1525, -18.0478,  ...,   6.6565, -10.9533,  -9.5993],
           [  5.1988,   9.7145,  -8.7255,  ...,   1.7275,   6.2185, -15.1953],
           ...,
           [ -0.8127,   0.4938,   2.9823,  ...,  -7.6766,   4.4492, -10.5318],
           [ 17.7455,   4.2289,  -1.0179,  ...,   0.5332,   6.9129, -10.6715],
           [ -3.2642,  -1.1587,   0.9091,  ...,  12.3628,   3.0298,   8.1988]]],
         device='cuda:0')
  
  a = torch.randn(2, 100, 64).to(device)
  b = torch.randn(2, 64, 100).to(device)
  torch.matmul(a, b)
  
  Traceback (most recent call last):
    File &quot;/home/cide/anaconda3/envs/nmmo/lib/python3.9/code.py&quot;, line 90, in runcode
      exec(code, self.locals)
    File &quot;&lt;input&gt;&quot;, line 1, in &lt;module&gt;
  RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling `cublasSgemmStridedBatched( handle, opa, opb, m, n, k, &amp;alpha, a, lda, stridea, b, ldb, strideb, &amp;beta, c, ldc, stridec, num_batches)`
  
  a = torch.randn(200, 64).to(device)
  b = torch.randn(64, 200).to(device)
  torch.matmul(a, b)
  
  tensor([[  5.9775,  13.0003,   0.4235,  ...,   4.0292, -17.0294,  -7.5343],
          [  4.3818,  -6.7144,   3.4724,  ...,   6.9620,   2.7793,  16.6472],
          [  1.8819,  -4.0363,   7.1150,  ...,  -6.0632,  -7.7502,   7.5797],
          ...,
          [  1.6404,   8.4560,   8.4408,  ...,  -3.5046,   7.3649,  -7.1671],
          [  2.1508,  14.0800, -10.0840,  ...,  -5.2585,   5.2174, -11.6113],
          [ 14.9206,  -6.9602,  11.4180,  ...,   4.3933,  -9.2923,  -8.2359]],
         device='cuda:0')
  </code></pre>
  <p>It seems that the error is caused by the shortage of my GPU memory space. However, my GPU usage is less than 20%.</p>
  <pre><code>+-----------------------------------------------------------------------------+
  | NVIDIA-SMI 470.103.01   Driver Version: 470.103.01   CUDA Version: 11.4     |
  |-------------------------------+----------------------+----------------------+
  | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
  | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
  |                               |                      |               MIG M. |
  |===============================+======================+======================|
  |   0  NVIDIA TITAN Xp     Off  | 00000000:03:00.0  On |                  N/A |
  | 26%   45C    P8    15W / 250W |   2034MiB / 12192MiB |      8%      Default |
  |                               |                      |                  N/A |
  +-------------------------------+----------------------+----------------------+
  |   1  NVIDIA TITAN Xp     Off  | 00000000:04:00.0 Off |                  N/A |
  | 23%   40C    P8    11W / 250W |      4MiB / 12196MiB |      0%      Default |
  |                               |                      |                  N/A |
  +-------------------------------+----------------------+----------------------+
                                                                                 
  +-----------------------------------------------------------------------------+
  | Processes:                                                                  |
  |  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
  |        ID   ID                                                   Usage      |
  |=============================================================================|
  |    0   N/A  N/A      1271      G   /usr/lib/xorg/Xorg                203MiB |
  |    0   N/A  N/A      2400      G   /usr/bin/gnome-shell               38MiB |
  |    0   N/A  N/A      3652      G   ...AAAAAAAAA= --shared-files       13MiB |
  |    0   N/A  N/A     22114      C   ...nda3/envs/nmmo/bin/python      859MiB |
  |    0   N/A  N/A     22882      G   ..._22757.log --shared-files       37MiB |
  |    0   N/A  N/A     27069      C   ...nda3/envs/nmmo/bin/python      859MiB |
  |    0   N/A  N/A     28706      G   ...mviewer/tv_bin/TeamViewer       11MiB |
  |    0   N/A  N/A     29739      G   /usr/bin/nvidia-settings            3MiB |
  +-----------------------------------------------------------------------------+
  </code></pre>
  <p>Moreover, if the 3rd~8th line in the codes are not run, it won't crash.</p>