title: CUDA kernel failed : no kernel image is available for execution on the device, Error when running PyTorch model inside Google Compute VM
body: <p>I have a docker image of a PyTorch model that returns this error when run inside a google compute engine VM running on debian/Tesla P4 GPU/google deep learning image:</p>
  
  <pre><code>CUDA kernel failed : no kernel image is available for execution on the device
  </code></pre>
  
  <p>This occurs on the line where my model is called. The PyTorch model includes custom c++ extensions, I'm using this model <a href="https://github.com/daveredrum/Pointnet2.ScanNet" rel="nofollow noreferrer">https://github.com/daveredrum/Pointnet2.ScanNet</a></p>
  
  <p>My image installs these at runtime</p>
  
  <p>The image runs fine on my local system. Both VM and my system have these versions:</p>
  
  <p>Cuda compilation tools    10.1, V10.1.243</p>
  
  <p>torch                     1.4.0                 </p>
  
  <p>torchvision               0.5.0 </p>
  
  <p>The main difference is the GPU as far as I'm aware</p>
  
  <p>Local: </p>
  
  <pre><code>+-----------------------------------------------------------------------------+
  | NVIDIA-SMI 435.21       Driver Version: 435.21       CUDA Version: 10.1     |
  |-------------------------------+----------------------+----------------------+
  | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
  | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
  |===============================+======================+======================|
  |   0  GeForce GTX 960M    Off  | 00000000:01:00.0 Off |                  N/A |
  | N/A   36C    P8    N/A /  N/A |    361MiB /  2004MiB |      0%      Default |
  +-------------------------------+----------------------+----------------------+
  </code></pre>
  
  <p>VM:</p>
  
  <pre><code>+-----------------------------------------------------------------------------+
  | NVIDIA-SMI 418.87.01    Driver Version: 418.87.01    CUDA Version: 10.1     |
  |-------------------------------+----------------------+----------------------+
  | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
  | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
  |===============================+======================+======================|
  |   0  Tesla P4            Off  | 00000000:00:04.0 Off |                    0 |
  | N/A   42C    P0    23W /  75W |      0MiB /  7611MiB |      3%      Default |
  </code></pre>
  
  <p>If I ssh into the VM <code>torch.cuda.is_available()</code> returns <code>true</code></p>
  
  <p>Therefore I suspect it must be something to do with the compilation of the extensions</p>
  
  <p>This is the relevant part of my docker file:</p>
  
  <pre><code>ENV CUDA_HOME "/usr/local/cuda-10.1"
  ENV PATH /usr/local/nvidia/bin:/usr/local/cuda-10.1/bin:${PATH}
  ENV NVIDIA_VISIBLE_DEVICES all
  ENV NVIDIA_DRIVER_CAPABILITIES compute,utility
  ENV NVIDIA_REQUIRE_CUDA "cuda&gt;=10.1 brand=tesla,driver&gt;=384,driver&lt;385 brand=tesla,driver&gt;=396,driver&lt;397 brand=tesla,driver&gt;=410,driver&lt;411 brand=tesla,driver&gt;=418,driver&lt;419"
  ENV FORCE_CUDA=1
  
  # CUDA 10.1-specific steps
  RUN conda install -c open3d-admin open3d
  RUN conda install -y -c pytorch \
      cudatoolkit=10.1 \
      "pytorch=1.4.0=py3.6_cuda10.1.243_cudnn7.6.3_0" \
      "torchvision=0.5.0=py36_cu101" \
   &amp;&amp; conda clean -ya
  RUN pip install -r requirements.txt
  RUN pip install flask
  RUN pip install plyfile
  RUN pip install scipy
  
  
  # Install OpenCV3 Python bindings
  RUN sudo apt-get update &amp;&amp; sudo apt-get install -y --no-install-recommends \
      libgtk2.0-0 \
      libcanberra-gtk-module \
      libgl1-mesa-glx \
   &amp;&amp; sudo rm -rf /var/lib/apt/lists/*
  
  RUN dir
  RUN cd pointnet2 &amp;&amp; python setup.py install
  RUN cd ..
  </code></pre>
  
  <p>I have already re-running this line from ssh in the VM:</p>
  
  <pre><code>TORCH_CUDA_ARCH_LIST="6.0 6.1 7.0" python setup.py install
  </code></pre>
  
  <p>Which I think targets the installation to the Tesla P4 compute capability?</p>
  
  <p>Is there some other setting or troubleshooting step I can try?</p>
  
  <p>I didn't know anything about docker/VMs/pytorch extensions until a couple of days ago, so somewhat shooting in the dark. Also this is my first stackoverflow post, apologies if I'm not following some etiquette, feel free to point out.</p>