<p>Im new to Kubernetes, my goal is to create a serverless like architecture on GPUs (i.e fan out to 1000+ pods)</p>
<p>I understand a <a href="https://kubernetes.io/docs/concepts/architecture/nodes/" rel="nofollow noreferrer">node may be a virtual or physical machine</a>. I am using GKE to help manage k8s. My node machine config is <code>n1-standard-4</code> with <code>1 x NVIDIA Tesla T4</code>.</p>
<p>With that setup it seems I could only have 4 pods, if I wanted lets say 16 pods per node, I could use <code>n1-standard-16</code>.</p>
<p>Lets say we are using <code>n1-standard-4</code> and ran 4 pods on that node, how can we give each node access to the GPU? Currently I am only able to run one pod, while the other pods stay on pending. This seems to only happen when I add the gpu resource in my YAML file.</p>
<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/name: load-balancer-example
  name: hello-world
spec:
  replicas: 4
  selector:
    matchLabels:
      app.kubernetes.io/name: load-balancer-example
  template:
    metadata:
      labels:
        app.kubernetes.io/name: load-balancer-example
    spec:
      containers:
        - image: CUSTOM_IMAGE_WITH_NVIDIA/CUDA/UBUNTU
          name: test
          ports:
            - containerPort: 3000
          resources:
            limits:
              nvidia.com/gpu: 1
</code></pre>
<p>Without the GPU resource and with a basic node container it seems to fan out fine. With the GPU resource I can only get one POD to run.</p>

Michael Joseph Aubry
<p>What you are creating is not a <code>Pod</code> but a <code>Deployment</code> with a replica count of 4, which is essentially 4 pods. All 4 of these pods are using your <code>n1-standard-4</code> type of node.</p>
<p>There are certain limitations when it comes to using GPUs with pods. This is very different from CPU sharing. In short, GPUs are only supposed to be specified in the limits section, which means:</p>
<ol>
<li>You can specify GPU limits without specifying requests because Kubernetes will use the limit as the request value by default.</li>
<li>You can specify GPU in both limits and requests but these two values must be equal.</li>
<li>You cannot specify GPU requests without specifying limits.</li>
<li>Containers (and Pods) do not share GPUs. There's no overcommitting of GPUs.</li>
<li>Each container can request one or more GPUs. It is not possible to request a fraction of a GPU.</li>
</ol>
<p>You can read more about these limitations <a href="https://kubernetes.io/docs/tasks/manage-gpus/scheduling-gpus/#using-device-plugins" rel="nofollow noreferrer">here</a>.</p>
<p>Your best option, will be to create a <code>node pool</code> with your desired GPU type. This node pool will have # nodes = # pods in your deployment and each node will host only 1 pod, and will have 1 GPU of your choice. I suggest this instead of multiple GPUs/node because you want to have a fan-out/scale-out architecture, so more smaller nodes will be better than less larger nodes.</p>
<p>You can read more about how to do this on the GKE docs <a href="https://cloud.google.com/kubernetes-engine/docs/how-to/gpus" rel="nofollow noreferrer">here</a>.</p>
<blockquote>
<p>Note that having <code>n1-standard-4</code> doesn't mean you can have 4 pods on the node. It simply means the node has 4 vCPUs which you can share across as many pods as needed. But since you want to run GPU workloads, this node type should not matter much, as long as you attach the right amount of GPU resources.</p>
</blockquote>
