<p>I have setup a kubernetes node with a nvidia tesla k80 and followed <a href="https://cloud.google.com/kubernetes-engine/docs/how-to/gpus" rel="nofollow noreferrer">this tutorial</a> to try to run a pytorch docker image with nvidia drivers and cuda drivers working.</p>

<p>My nvidia drivers and cuda drivers are all accessible inside my pod at <code>/usr/local</code>:</p>

<pre><code>$&gt; ls /usr/local
bin  cuda  cuda-10.0  etc  games  include  lib  man  nvidia  sbin  share  src
</code></pre>

<p>And my GPU is also recongnized by my image <code>nvidia/cuda:10.0-runtime-ubuntu18.04</code>:</p>

<pre><code>$&gt; /usr/local/nvidia/bin/nvidia-smi
Fri Nov  8 16:24:35 2019
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.79       Driver Version: 410.79       CUDA Version: 10.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K80           Off  | 00000000:00:04.0 Off |                    0 |
| N/A   73C    P8    35W / 149W |      0MiB / 11441MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+
</code></pre>

<p>But after installing pytorch 1.3.0 i'm not able to make pytorch recognize my cuda installation even with <code>LD_LIBRARY_PATH</code> set to <code>/usr/local/nvidia/lib64:/usr/local/cuda/lib64</code>:</p>

<pre><code>$&gt; python3 -c "import torch; print(torch.cuda.is_available())"
False

$&gt; python3
Python 3.6.8 (default, Oct  7 2019, 12:59:55)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
&gt;&gt;&gt; import torch
&gt;&gt;&gt; print ('\t\ttorch.cuda.current_device()    =', torch.cuda.current_device())
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "/usr/local/lib/python3.6/dist-packages/torch/cuda/__init__.py", line 386, in current_device
    _lazy_init()
  File "/usr/local/lib/python3.6/dist-packages/torch/cuda/__init__.py", line 192, in _lazy_init
    _check_driver()
  File "/usr/local/lib/python3.6/dist-packages/torch/cuda/__init__.py", line 111, in _check_driver
    of the CUDA driver.""".format(str(torch._C._cuda_getDriverVersion())))
AssertionError:
The NVIDIA driver on your system is too old (found version 10000).
Please update your GPU driver by downloading and installing a new
version from the URL: http://www.nvidia.com/Download/index.aspx
Alternatively, go to: https://pytorch.org to install
a PyTorch version that has been compiled with your version
of the CUDA driver.
</code></pre>

<p>The error above is strange because my cuda version for my image is 10.0 and Google GKE mentions that:</p>

<blockquote>
  <p>The latest supported CUDA version is 10.0</p>
</blockquote>

<p>Also, it's GKE's daemonsets that automatically installs NVIDIA drivers</p>

<blockquote>
  <p>After adding GPU nodes to your cluster, you need to install NVIDIA's device drivers to the nodes.</p>
  
  <p>Google provides a DaemonSet that automatically installs the drivers for you.
  Refer to the section below for installation instructions for Container-Optimized OS (COS) and Ubuntu nodes.</p>
  
  <p>To deploy the installation DaemonSet, run the following command:
  kubectl apply -f <a href="https://raw.githubusercontent.com/GoogleCloudPlatform/container-engine-accelerators/master/nvidia-driver-installer/cos/daemonset-preloaded.yaml" rel="nofollow noreferrer">https://raw.githubusercontent.com/GoogleCloudPlatform/container-engine-accelerators/master/nvidia-driver-installer/cos/daemonset-preloaded.yaml</a></p>
</blockquote>

<p>I have tried everything i could think of, without success...</p>

lifeguru42
<p>I have resolved my problem by downgrading my pytorch version by buildling my docker images from <code>pytorch/pytorch:1.2-cuda10.0-cudnn7-devel</code>.</p>

<p>I still don't really know why before it was not working as it should otherwise then by guessing that <code>pytorch 1.3.0</code> is not compatible with <code>cuda 10.0</code>.</p>
