title: Kubernetes nvidia-gpu UnexpectedAdmissionError
body: <p>I have a k8s cluster with one master and one worker. Worker has GPUs, and I am able to schedule pods on it using GPUs.</p>
  
  <p>I add the second worker, also with GPUs. Everything is properly installed on that second worker. When I try to schedule GPU busybox on that second worker I get status <code>UnexpectedAdmissionError</code>.</p>
  
  <p>With <code>kubectl describe pod busybox</code> I see <code>Warning  UnexpectedAdmissionError  7m    kubelet, wikiserver  Update plugin resources failed due to requested number of devices unavailable for nvidia.com/gpu. Requested: 1, Available: 0, which is unexpected.</code> which is odd since with <code>kubectl describe nodes second-worker</code> I see that both Capacity and Allocatable are <code>nvidia.com/gpu: 1</code></p>
  
  <p>I was able to successfully schedule GPU busybox on first worker. Also, I was able to schedule CPU busybox on both workers.</p>
  
  <p>This is yaml conf for GPU busybox:</p>
  
  <pre><code>apiVersion: v1
  kind: Pod
  metadata:
    name: busybox
    namespace: default
  spec:
    containers:
    - image: busybox
      command:
        - sleep
        - "3600"
      imagePullPolicy: IfNotPresent
      name: busybox
      resources:
          limits:
            nvidia.com/gpu: 1 
    restartPolicy: Always
    nodeName: secondworker
  
  </code></pre>
  
  <p>The second worker (and first worker and the master) is configured as described by <a href="https://github.com/NVIDIA/k8s-device-plugin" rel="nofollow noreferrer">nvidia-device-plugin</a>.</p>