<p>I've got a kubeflow k8s cluster with custom GPU-powered preemptible node pool at <code>us-central1-a</code>:
<a href="https://i.stack.imgur.com/3furl.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/3furl.png" alt="enter image description here"></a></p>

<p>I run a kubeflow notebook server on these GPU nodes.
By some mysterious reason nodes get <code>compute.instances.preempted</code> message very soon after start (5-10 minutes):
<a href="https://i.stack.imgur.com/uonhr.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/uonhr.png" alt="enter image description here"></a></p>

<p>Why is this happening?</p>

orkenstein
<p>Since you have created a pool of preemptible nodes, this is pretty much expected behavior.  GCE can terminate preemptible instances <a href="https://cloud.google.com/compute/docs/instances/preemptible#limitations" rel="nofollow noreferrer">at any time</a>, and the only real guarantee you have is that you won't be charged for the instance (but you will be charged for any requested premium OS -- of which COS is not one) if they run for less than a minute (and, of course, that they will always be preempted after 24 hours).</p>

<p>GPU nodes are likely to be in high demand, and as with other preemptible instances this will be subject to the particular zone and time of day.  If you need the instances to stay available, you should use full price instances.  Using GKE, there is a way to <a href="https://cloud.google.com/blog/products/gcp/accelerate-highly-parallelized-compute-tasks-with-gpus-in-kubernetes-engine" rel="nofollow noreferrer">autoscale GPU nodes</a> to help control costs.</p>
