<p>I'm trying to setup Kubernetes with Nvidia GPU nodes/slaves.
I followed the guide at <a href="https://docs.nvidia.com/datacenter/kubernetes-install-guide/index.html" rel="nofollow noreferrer">https://docs.nvidia.com/datacenter/kubernetes-install-guide/index.html</a> and I was able to get the node join the cluster. I tried the below kubeadm example pod:</p>

<pre><code>apiVersion: v1
kind: Pod
metadata:
  name: gpu-pod
spec:
  containers:
    - name: cuda-container
      image: nvidia/cuda:9.0-base
      command: ["sleep"]
      args: ["100000"]
      extendedResourceRequests: ["nvidia-gpu"]
  extendedResources:
    - name: "nvidia-gpu"
      resources:
        limits:
          nvidia.com/gpu: 1
      affinity:
        required:
          - key: "nvidia.com/gpu-memory"
            operator: "Gt"
            values: ["8000"]
</code></pre>

<p>The pod fails scheduling &amp; the kubectl events shows:</p>

<pre><code>4s          2m           14        gpu-pod.15487ec0ea0a1882        Pod                                          Warning   FailedScheduling        default-scheduler            0/2 nodes are available: 1 Insufficient nvidia.com/gpu, 1 PodToleratesNodeTaints.
</code></pre>

<p>I'm using AWS EC2 instances. m5.large for the master node &amp; g2.8xlarge for the slave node. Describing the node also gives "<strong>nvidia.com/gpu:  4</strong>".
Can anybody help me out if I'm missing any steps/configurations?</p>

<p>According to the AWS G2 <a href="https://aws.amazon.com/blogs/aws/new-g2-instance-type-with-4x-more-gpu-power/" rel="nofollow noreferrer">documentation</a>, <code>g2.8xlarge</code> servers have the following resources:</p>

<ul>
<li>Four NVIDIA GRID GPUs, each with 1,536 CUDA cores and 4 GB of video
memory and the ability to encode either four real-time HD video
streams at 1080p or eight real-time HD video streams at 720P. </li>
<li>32 vCPUs. </li>
<li>60 GiB of memory.</li>
<li>240 GB (2 x 120) of SSD storage.</li>
</ul>

<p>Looking at the comments, 60 GB is standard RAM, and it is used for regular calculations. <code>g2.8xlarge</code> servers have 4 GPUs with 4 GB of GPU memory each, and this memory is used for calculations in <code>nvidia/cuda</code> containers. </p>

<p>In your case, it is requested 8 GB of GPU memory per GPU, but your server has only 4 GB. Therefore, the cluster experiences a lack of resources for scheduling the POD. So, try to reduce the memory usage in the Pod settings or try to use a server with a larger amount of GPU memory.</p>
