<p>I am trying to understand the K8s gpu practices better, and implementing a small K8s GPU cluster which is suppose to work like below.</p>

<p>This going to be little long explanation, but I hope it will help to have many questions at once place to understand GPU practices better in Kubernetes.</p>

<h2>Application Requirement</h2>

<ul>
<li>I want to create a K8s autoscale cluster. </li>
<li>Pods are running the models say a tensorflow based deep learning program.  </li>
<li>Pods are waiting for a message in pub sub queue to appear and it can proceed
its execution once it recieves a message. </li>
<li>Now a message is queued in a PUB/SUB queue. </li>
<li>As message is available, pods reads it and execute deep learning program.</li>
</ul>

<h2>Cluster requirement</h2>

<p>If no message is present in queue and none of the GPU based pods are executing program( i mean not using gpu), then gpu node pool should scale down to 0.</p>

<h2>Design 1</h2>

<p>Create a gpu node pool. Each node contains N GPU, where N >= 1.
Assign model trainer pod to each gpu. That is 1:1 mapping of pods and GPU.
When I tried assigning 2 pods to 2 GPU machine where each pod is suppose to run a mnist program.</p>

<p>What I noticed is  </p>

<p>1 pod got allocated and executes the program and later it went into crash loop. May be I am doing some mistake as my docker image is suppose to run program once only as I was just doing feasibility test of running 2 pods simultaneously on 2 gpu of same node.Below is the error</p>

<pre> Message   Reason  First Seen  Last Seen   Count
Back-off restarting failed container    BackOff Jun 21, 2018, 3:18:15 PM    Jun 21, 2018, 4:16:42 PM    143
pulling image "nkumar15/mnist"  Pulling Jun 21, 2018, 3:11:33 PM    Jun 21, 2018, 3:24:52 PM    5
Successfully pulled image "nkumar15/mnist"  Pulled  Jun 21, 2018, 3:12:46 PM    Jun 21, 2018, 3:24:52 PM    5
Created container   Created Jun 21, 2018, 3:12:46 PM    Jun 21, 2018, 3:24:52 PM    5
Started container   Started Jun 21, 2018, 3:12:46 PM    Jun 21, 2018, 3:24:52 PM    5
</pre>

<p>The other pod didn't get assigned at all to GPU. Below is the message from pod events</p>

<p><strong>0/3 nodes are available: 3 Insufficient nvidia.com/gpu.</strong></p>

<h2>Design 2</h2>

<p>Have multiple GPU machines in gpu node pool with each node having only 1 GPU.<br>
K8s, will assign each pod to each available GPU in node and hopefully there won't be any issue. I am yet to try this.</p>

<h2>Questions</h2>

<ol>
<li>Is there any suggested practice to design above type of system in kubernetes as of version 1.10?</li>
<li>Is Design 1 approach not feasible as of 1.10 release? For eg, I have 2 GPU node with 24 GB GPU memory, is it possible such that K8s assign 
1 pod to each GPU and each pods execute its own workload with 12GB memory limit each?</li>
<li>How do I scale down gpu node pool to 0 size through autoscaler?</li>
<li>In Design 2, say what if I run out of GPU memory? as curently in GCP 1 GPU node doesn't have more than 16 GB memory.</li>
</ol>

<p>Again apologies for such a long question, but I hope it will help other also.</p>

<h2>Updates</h2>

<p><strong>For question 2</strong>
I created a new cluster to reproduce same issue which I faced multiple times before, I am not sure what changed this time but 2nd pod is successfully allocated a GPU. I think with this result I can confirm that 1gpu-1pod mapping is allowed in a multi gpu single node
However restricting memory per gpu process is not feasible as of 1.10.</p>

<p>Both designs are supported in 1.10. I view design 2 as a special case of 1. You don't necessarily need to have 1 GPU per node. In case your pod needs more GPUs and memory, you have to have multiple GPUs per node, as you mentioned in question (4). I'd go with 1 unless there's a reason not to.</p>

<p>I think the best practice would be create a new cluster with no GPUs (a cluster has a default node pool), and then create a GPU node pool and attach it to the cluster. Your non-GPU workload can run in the default pool, and the GPU workload can run in the GPU pool. To support scaling-down to 0 GPUs, you need to set <code>--num-nodes</code> and <code>--min-nodes</code> to be 0 when creating the GPU node pool.</p>

<p>Docs:</p>

<p><em>Create a cluster with no GPUs: <a href="https://cloud.google.com/kubernetes-engine/docs/how-to/creating-a-cluster#creating_a_cluster" rel="nofollow noreferrer">https://cloud.google.com/kubernetes-engine/docs/how-to/creating-a-cluster#creating_a_cluster</a></em></p>

<p><em>Create a GPU node pool for an existing cluster: <a href="https://cloud.google.com/kubernetes-engine/docs/concepts/gpus#gpu_pool" rel="nofollow noreferrer">https://cloud.google.com/kubernetes-engine/docs/concepts/gpus#gpu_pool</a></em></p>
