<p>I created a small cluster with GPU nodes on GKE like so:</p>

<pre class="lang-sh prettyprint-override"><code># create cluster and CPU nodes
gcloud container clusters create clic-cluster \
    --zone us-west1-b \
    --machine-type n1-standard-1 \
    --enable-autoscaling \
    --min-nodes 1 \
    --max-nodes 3 \
    --num-nodes 2

# add GPU nodes
gcloud container node-pools create gpu-pool \
    --zone us-west1-b \
    --machine-type n1-standard-2 \
    --accelerator type=nvidia-tesla-k80,count=1 \
    --cluster clic-cluster \
    --enable-autoscaling \
    --min-nodes 1 \
    --max-nodes 2 \
    --num-nodes 1
</code></pre>

<p>When I submit a GPU job it successfully ends up running on the GPU node. However, when I submit a second job I get an <code>UnexpectedAdmissionError</code> from kubernetes:</p>

<blockquote>
  <p>Update plugin resources failed due to requested number of devices
  unavailable for nvidia.com/gpu. Requested: 1, Available: 0, which is
  unexpected.</p>
</blockquote>

<p>I would have expected the cluster to start the second GPU node and place the job there. Any idea why this didn't happen? My job spec looks roughly like this:</p>

<pre><code>apiVersion: batch/v1
kind: Job
metadata:
  name: &lt;job_name&gt;
spec:
  template:
    spec:
      initContainers:
      - name: decode
        image: "&lt;decoder_image&gt;"
        resources:
          limits:
            nvidia.com/gpu: 1
        command: [...]
     [...]     
     containers:
      - name: evaluate
        image: "&lt;evaluation_image&gt;"
        command: [...]
</code></pre>

<p>The resource constraint needs to be added to the <code>containers</code> spec as well:</p>

<pre><code>piVersion: batch/v1
kind: Job
metadata:
  name: &lt;job_name&gt;
spec:
  template:
    spec:
      initContainers:
      - name: decode
        image: "&lt;decoder_image&gt;"
        resources:
          limits:
            nvidia.com/gpu: 1
        command: [...]
     [...]     
     containers:
      - name: evaluate
        image: "&lt;evaluation_image&gt;"
        resources:
          limits:
            nvidia.com/gpu: 1
        command: [...]
</code></pre>

<p>I only required a GPU in one of the <code>initContainers</code>, but this seems to confuse the scheduler. Now autoscaling and scheduling works as expected.</p>
