<p>I got a K8S+DinD issue:</p>
<ul>
<li>launch Kubernetes cluster</li>
<li>start a main docker image and a DinD image inside this cluster</li>
<li>when running a job requesting GPU, got error <code>could not select device driver &quot;nvidia&quot; with capabilities: [[gpu]]</code></li>
</ul>
<p>Full error</p>
<pre><code>http://localhost:2375/v1.40/containers/long-hash-string/start: Internal Server Error (&quot;could not select device driver &quot;nvidia&quot; with capabilities: [[gpu]]&quot;)
</code></pre>
<p><code>exec</code> to the DinD image inside of K8S pod, <code>nvidia-smi</code> is not available.</p>
<p>Some debugging and it seems it's due to the DinD is missing the Nvidia-docker-toolkit, I had the same error when I ran the same job directly on my local laptop docker, I fixed the same error by installing <strong>nvidia-docker2</strong> <code>sudo apt-get install -y nvidia-docker2</code>.</p>
<p>I'm thinking maybe I can try to install nvidia-docker2 to the DinD 19.03 (docker:19.03-dind), but not sure how to do it? By multiple stage docker build?</p>
<p>Thank you very much!</p>
<hr />
<p>update:</p>
<p>pod spec:</p>
<pre><code>spec:
    containers:
      - name: dind-daemon
        image: docker:19.03-dind
</code></pre>

<p>I got it working myself.</p>
<p>Referring to</p>
<ul>
<li><a href="https://github.com/NVIDIA/nvidia-docker/issues/375" rel="nofollow noreferrer">https://github.com/NVIDIA/nvidia-docker/issues/375</a></li>
<li><a href="https://github.com/Henderake/dind-nvidia-docker" rel="nofollow noreferrer">https://github.com/Henderake/dind-nvidia-docker</a></li>
</ul>
<blockquote>
<p>First, I modified the ubuntu-dind image (<a href="https://github.com/billyteves/ubuntu-dind" rel="nofollow noreferrer">https://github.com/billyteves/ubuntu-dind</a>) to install nvidia-docker (i.e. added the instructions in the nvidia-docker site to the Dockerfile) and changed it to be based on nvidia/cuda:9.2-runtime-ubuntu16.04.</p>
</blockquote>
<blockquote>
<p>Then I created a pod with two containers, a frontend ubuntu container and the a privileged docker daemon container as a sidecar. The sidecar's image is the modified one I mentioned above.</p>
</blockquote>
<p>But since this post is 3 year ago from now, I did spent quite some time to match up the dependencies versions, repo migration over 3 years, etc.</p>
<p>My modified version of Dockerfile to build it</p>
<pre><code>ARG CUDA_IMAGE=nvidia/cuda:11.0.3-runtime-ubuntu20.04
FROM ${CUDA_IMAGE}

ARG DOCKER_CE_VERSION=5:18.09.1~3-0~ubuntu-xenial


RUN apt-get update -q &amp;&amp; \
    apt-get install -yq \
        apt-transport-https \
        ca-certificates \
        curl \
        gnupg-agent \
        software-properties-common &amp;&amp; \
    curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add - &amp;&amp; \
    add-apt-repository \
       &quot;deb [arch=amd64] https://download.docker.com/linux/ubuntu \
       $(lsb_release -cs) \
       stable&quot;  &amp;&amp; \
    apt-get update -q &amp;&amp; apt-get install -yq docker-ce docker-ce-cli containerd.io

# https://github.com/docker/docker/blob/master/project/PACKAGERS.md#runtime-dependencies
RUN set -eux; \
    apt-get update -q &amp;&amp; \
    apt-get install -yq \
        btrfs-progs \
        e2fsprogs \
        iptables \
        xfsprogs \
        xz-utils \
# pigz: https://github.com/moby/moby/pull/35697 (faster gzip implementation)
        pigz \
#        zfs \
        wget


# set up subuid/subgid so that &quot;--userns-remap=default&quot; works out-of-the-box
RUN set -x \
    &amp;&amp; addgroup --system dockremap \
    &amp;&amp; adduser --system -ingroup dockremap dockremap \
    &amp;&amp; echo 'dockremap:165536:65536' &gt;&gt; /etc/subuid \
    &amp;&amp; echo 'dockremap:165536:65536' &gt;&gt; /etc/subgid

# https://github.com/docker/docker/tree/master/hack/dind
ENV DIND_COMMIT 37498f009d8bf25fbb6199e8ccd34bed84f2874b

RUN set -eux; \
    wget -O /usr/local/bin/dind &quot;https://raw.githubusercontent.com/docker/docker/${DIND_COMMIT}/hack/dind&quot;; \
    chmod +x /usr/local/bin/dind


##### Install nvidia docker #####
# Add the package repositories
RUN curl -fsSL https://nvidia.github.io/nvidia-docker/gpgkey | apt-key add --no-tty -

RUN distribution=$(. /etc/os-release;echo $ID$VERSION_ID) &amp;&amp; \
    echo $distribution &amp;&amp;  \
    curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | \
      tee /etc/apt/sources.list.d/nvidia-docker.list

RUN apt-get update -qq --fix-missing

RUN apt-get install -yq nvidia-docker2

RUN sed -i '2i \ \ \ \ &quot;default-runtime&quot;: &quot;nvidia&quot;,' /etc/docker/daemon.json

RUN mkdir -p /usr/local/bin/
COPY dockerd-entrypoint.sh /usr/local/bin/
RUN chmod 777 /usr/local/bin/dockerd-entrypoint.sh
RUN ln -s /usr/local/bin/dockerd-entrypoint.sh /

VOLUME /var/lib/docker
EXPOSE 2375

ENTRYPOINT [&quot;dockerd-entrypoint.sh&quot;]
#ENTRYPOINT [&quot;/bin/sh&quot;, &quot;/shared/dockerd-entrypoint.sh&quot;]
CMD []

</code></pre>
<p>When I use <code>exec</code> to login into the Docker-in-Docker container, I can successfully run <code>nvidia-smi</code> (which previously return not found error then cannot run any GPU resource related docker run)</p>
<p>Welcome to pull my image at <code>brandsight/dind:nvidia-docker</code></p>
