<p>I have a cluster running on GCP that currently consists entirely of preemtible nodes. We're experiencing issues where kube-dns becomes unavailable (presumably because a node has been preempted). We'd like to improve the resilience of DNS by moving <code>kube-dns</code> pods to more stable nodes.</p>

<p>Is it possible to schedule system cluster critical pods like <code>kube-dns</code> (or all pods in the <code>kube-system</code> namespace) on a node pool of only non-preemptible nodes? I'm wary of using affinity or anti-affinity or taints, since these pods are auto-created at cluster bootstrapping and any changes made could be clobbered by a Kubernetes version upgrade. Is there a way do do this that will persist across upgrades?</p>

<p>The solution was to use taints and tolerations in conjunction with node affinity. We created a second node pool, and added a taint to the preemptible pool.</p>

<p>Terraform config:</p>

<pre><code>resource "google_container_node_pool" "preemptible_worker_pool" {
  node_config {
    ...
    preemptible     = true

    labels {
      preemptible = "true"
      dedicated   = "preemptible-worker-pool"
    }

    taint {
      key    = "dedicated"
      value  = "preemptible-worker-pool"
      effect = "NO_SCHEDULE"
    }
  }
}
</code></pre>

<p>We then used a <code>toleration</code> and <code>nodeAffinity</code> to allow our existing workloads to run on the tainted node pool, effectively forcing the cluster-critical pods to run on the untainted (non-preemtible) node pool.</p>

<p>Kubernetes config:</p>

<pre><code>spec:
  template:
    spec:
      # The affinity + tolerations sections together allow and enforce that the workers are
      # run on dedicated nodes tainted with "dedicated=preemptible-worker-pool:NoSchedule".
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: dedicated
                operator: In
                values:
                - preemptible-worker-pool
      tolerations:
      - key: dedicated
        operator: "Equal"
        value: preemptible-worker-pool
        effect: "NoSchedule"
</code></pre>
