<p>i need a solution for that:</p>

<p>i have 2 node pools in gcloud kubernetes, first is preemptible and autoscaling, second is only autoscaling.</p>

<p>Jobs should be started on the first one ( with preemptible VMs ), but when no resources on the first pool are available Jobs should be started on the second one.</p>

<p>How can i realize that, maybe with Taints and Tolerations?</p>

<p>I don't think you can get exactly what you want with the Cluster Autoscaler but I'll hopefully give you a couple options and pointers to further explore.</p>

<ul>
<li>The Cluster Autoscaler has the notion of <a href="https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/FAQ.md#what-are-expanders" rel="noreferrer">Expanders</a> which can help determine which node group to scale up when a scaling event happens. The <code>price</code> expander seems to be close to what you want, but based on the <a href="https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/proposals/pricing.md" rel="noreferrer">description</a> of it, it doesn't look like it has support for preemptible VMs yet. You could explore that further, and possibly <a href="https://github.com/kubernetes/autoscaler/issues" rel="noreferrer">submit a feature request</a> to add support for preemptible node pools.</li>
<li><p>When choosing a mixture of preemptible and non-preemptible nodes, whenever there is a stock out on GCP and preemptible nodes are not available, it's very likely that non-preemptible nodes will <strong>also</strong> not be available. In that case you may find yourself with a small number of non-preemptible nodes in the cluster and without the ability to create new ones.</p>

<p>It may be a better idea to have a fixed minimum size of non-preemptible nodes, and auto-scale a preemptible node pool on top of that using the Cluster Autoscaler.</p></li>
</ul>
