title: Pycharm debugging using docker with GPUs
body: <h1>The Goal:</h1>
  
  <p>To debug a Python application in PyCharm, where I set the interpreter to a custom docker image, using Tensorflow and so requiring a GPU. The problem is that PyCharm's <em>command-building</em> doesn't offer a way to discover available GPUs, as far as I can tell.</p>
  
  <h1>Terminal - it works:</h1>
  
  <p>Enter a container with the following command, specifying which GPUs to make available (<code>--gpus</code>):</p>
  
  <pre><code>docker run -it --rm --gpus=all --entrypoint="/bin/bash" 3b6d609a5189        # image has an entrypoint, so I overwrite it
  </code></pre>
  
  <p>Inside the container, I can run <code>nvidia-smi</code> to see a GPU is found, and confirm Tensorflow finds it, using:</p>
  
  <pre><code>from tensorflow.python.client import device_lib
  device_lib.list_local_devices()
  # physical_device_desc: "device: 0, name: Quadro P2000, pci bus id: 0000:01:00.0, compute capability: 6.1"]
  </code></pre>
  
  <p>If I don't use the <code>--gpus</code> flag, no GPUs are discovered, as expected.
  Note: using docker version 19.03 and above, Nvidia runtimes are supports natively, so there is no need for <code>nvidia-docker</code> and also, the docker-run argument <code>--runtime=nvidia</code> is also deprecated. <a href="https://stackoverflow.com/a/58432877/3126298">Relevant thread</a>.</p>
  
  <h1>PyCharm - it doesn't work</h1>
  
  <p>Here is the configuration for the run:</p>
  
  <p><a href="https://i.stack.imgur.com/e1hI7.png" rel="noreferrer"><img src="https://i.stack.imgur.com/e1hI7.png" alt="configuration"></a></p>
  
  <p><em>(I realise some of those paths might look incorrect, but that isn't an issue for now)</em></p>
  
  <p>I set the interpreter to point to the same docker image and run the Python script, set a custom <code>LD_LIBRARY_PATH</code> as an argument to the run that matches where the <code>libcuda.so</code> is <code>locate</code>d in the docker image (I found it interactively inside a running container), but still no device is found:</p>
  
  <p><a href="https://i.stack.imgur.com/ri8z8.png" rel="noreferrer"><img src="https://i.stack.imgur.com/ri8z8.png" alt="error message"></a></p>
  
  <p>The error message shows the the CUDA library was able to be loaded (i.e. is was found on that <code>LD_LIBRARY_PATH</code>), but the device was still not found. This is why I believe the docker run argument <code>--gpus=all</code> must be set somewhere. I can't find a way to do that in PyCharm.</p>
  
  <h1>Other things I have tried:</h1>
  
  <ol>
  <li>In PyCharm, using a Docker execution template config (instead of a Python template) where it is possible to specify <em>run arguments</em>, so I hoped to pass <code>--gpus=all</code>, but that seems not to be supported by the parser of those options:</li>
  </ol>
  
  <p><a href="https://i.stack.imgur.com/yGXLD.png" rel="noreferrer"><img src="https://i.stack.imgur.com/yGXLD.png" alt="parse error"></a></p>
  
  <ol start="2">
  <li>I tried to set the default runtime to be <code>nvidia</code> in the docker daemon by including the following config in <code>/etc/docker/daemon.json</code>:</li>
  </ol>
  
  <blockquote>
  <pre><code>{
      "runtimes": {
          "nvidia": {
              "runtimeArgs": ["gpus=all"]
          }
      }
  }
  </code></pre>
  </blockquote>
  
  <p>I am not sure of the correct format for this, however. I have tried a few variants of the above, but nothing got the GPUs recognised. The example above could at least be parsed and allow me to restart the docker daemon without errors.</p>
  
  <ol start="3">
  <li><p>I noticed in the official Tensorflow docker images, they install a package (via <code>apt install</code>) called <code>nvinfer-runtime-trt-repo-ubuntu1804-5.0.2-ga-cuda10.0</code>, which sounds like a great tool, albeit seemingly just for TensorRT. I added it to my Dockerfile as a shot in the dark, but unfortunately it did not fix the issue.</p></li>
  <li><p>Adding <code>NVIDIA_VISIBLE_DEVICES=all</code> etc. to the environment variables of the PyCharm configuration, with no luck.</p></li>
  </ol>
  
  <p>I am using Python 3.6, PyCharm Professional 2019.3 and Docker 19.03.</p>