<p>we <a href="https://cloud.google.com/kubernetes-engine/docs/how-to/gpus#cos" rel="noreferrer">followed this guide</a> to use GPU enabled nodes in our existing cluster but when we try to schedule pods we're getting <code>2 Insufficient nvidia.com/gpu error</code></p>
<p><strong>Details:</strong></p>
<p>We are trying to use GPU in our existing cluster and for that we're able to successfully create a NodePool with a single node having GPU enabled.</p>
<p>Then as a next step according to the guide above we've to create a daemonset and we're also able to run the DS successfully.</p>
<p>But now when we are trying to schedule the Pod using the following resource section the pod becomes un-schedulable with this error <code>2 insufficient nvidia.com/gpu</code></p>
<pre><code>    resources:
      limits:
        nvidia.com/gpu: &quot;1&quot;
      requests:
        cpu: 200m
        memory: 3Gi
</code></pre>
<p><strong>Specs:</strong></p>
<pre><code>Node version - v1.18.17-gke.700 (+ v1.17.17-gke.6000) tried on both
Instance type - n1-standard-4
image - cos
GPU - NVIDIA Tesla T4
</code></pre>
<p>any help or pointers to debug this further will be highly appreaciated.</p>
<p>TIA,</p>
<hr />
<p>output of <code>kubectl get node &lt;gpu-node&gt; -o yaml</code> [Redacted]</p>
<pre><code>apiVersion: v1
kind: Node
metadata:
  labels:
    beta.kubernetes.io/arch: amd64
    beta.kubernetes.io/instance-type: n1-standard-4
    beta.kubernetes.io/os: linux
    cloud.google.com/gke-accelerator: nvidia-tesla-t4
    cloud.google.com/gke-boot-disk: pd-standard
    cloud.google.com/gke-container-runtime: docker
    cloud.google.com/gke-nodepool: gpu-node
    cloud.google.com/gke-os-distribution: cos
    cloud.google.com/machine-family: n1
    failure-domain.beta.kubernetes.io/region: us-central1
    failure-domain.beta.kubernetes.io/zone: us-central1-b
    kubernetes.io/arch: amd64
    kubernetes.io/os: linux
    node.kubernetes.io/instance-type: n1-standard-4
    topology.kubernetes.io/region: us-central1
    topology.kubernetes.io/zone: us-central1-b
  name: gke-gpu-node-d6ddf1f6-0d7j
spec:
  taints:
  - effect: NoSchedule
    key: nvidia.com/gpu
    value: present
status:
  ...
  allocatable:
    attachable-volumes-gce-pd: &quot;127&quot;
    cpu: 3920m
    ephemeral-storage: &quot;133948343114&quot;
    hugepages-2Mi: &quot;0&quot;
    memory: 12670032Ki
    pods: &quot;110&quot;
  capacity:
    attachable-volumes-gce-pd: &quot;127&quot;
    cpu: &quot;4&quot;
    ephemeral-storage: 253696108Ki
    hugepages-2Mi: &quot;0&quot;
    memory: 15369296Ki
    pods: &quot;110&quot;
  conditions:
    ...
  nodeInfo:
    architecture: amd64
    containerRuntimeVersion: docker://19.3.14
    kernelVersion: 5.4.89+
    kubeProxyVersion: v1.18.17-gke.700
    kubeletVersion: v1.18.17-gke.700
    operatingSystem: linux
    osImage: Container-Optimized OS from Google
</code></pre>
<p>Tolerations from the deployments</p>
<pre><code>  tolerations:
  - effect: NoSchedule
    key: nvidia.com/gpu
    operator: Exists
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
</code></pre>

<p>The <code>nvidia-gpu-device-plugin</code> should be installed in the GPU node as well. You should see <code>nvidia-gpu-device-plugin</code> DaemonSet in your <code>kube-system</code> namespace.</p>
<p>It should be automatically deployed by Google, but if you want to deploy it on your own, run the following command: <code>kubectl apply -f https://raw.githubusercontent.com/kubernetes/kubernetes/master/cluster/addons/device-plugins/nvidia-gpu/daemonset.yaml</code></p>
<p>It will install the GPU plugin in the node and afterwards your pods will be able to consume it.</p>
