<p>I have run into an issue where <code>helm install</code>ing my charts will work fine, but when I go to restart the system, the nvidia gpu operator will fail to validate.</p>
<p>Bootstrapping is simple:</p>
<p><code>$ microk8s enable gpu</code></p>
<p>&lt; watching dashboard for all the pods to turn green &gt;</p>
<p><code>$ microk8s helm install -n morpheus morpheus-ai-engine morpheus-ai-engine</code></p>
<p>&lt; watching for the morpheus pods to turn green &gt;</p>
<p>Now I can check if the <code>ai-engine</code> pod has GPU access:</p>
<pre><code>$ kubectl exec ai-engine-897d65cff-b2trz -- nvidia-smi
Wed Feb 22 16:35:32 2023       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.78.01    Driver Version: 525.78.01    CUDA Version: 12.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Quadro P400         Off  | 00000000:04:00.0 Off |                  N/A |
|  0%   38C    P8    N/A /  30W |     98MiB /  2048MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
+-----------------------------------------------------------------------------+
</code></pre>
<p>Running the test vector-add pod returns a <code>Test PASSED</code>.</p>
<p>The trouble comes when I restart microk8s. The <code>nvidia-device-plugin-validator</code> pod fails to load with an <code>UnexpectedAdmissionError</code> claiming that no GPUs are available. And running <code>nvidia-smi</code> in the <code>ai-engine</code> pod returns a &quot;command not found&quot;. The vector-add test pod won't start due to insufficient GPUs.</p>
<p>But if I uninstall the <code>ai-engine</code> chart and restart microk8s (waiting for the gpu operator pods to all turn green), I can then reinstall <code>ai-engine</code> and it works fine again, as does the vector-add test.</p>

<p>This is an issue I am comming across too which lead me hear, it looks like it was just recently fixed with this patch <a href="https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/release-notes.html#id2" rel="nofollow noreferrer">https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/release-notes.html#id2</a></p>
<p>Which will evict pods requesting gpus while the operator starts up again.
This should solve your issue as it did mine.</p>
