<p>I have a multi-regional testing setup on GKE k8s 1.9.4. Every cluster has:</p>

<ul>
<li>an ingress, configured with <code>kubemci</code></li>
<li>3 node pools with different node labels:

<ul>
<li>default-pool <code>system</code> (1vCPU / 2GB RAM)</li>
<li>frontend-pool <code>frontend</code> (2vCPU / 2GB RAM)</li>
<li>backend-pool <code>backend</code> (1vCPU / 600Mb RAM)</li>
</ul></li>
<li>HPA with scaling by the custom metric</li>
</ul>

<p>So stuff like <code>prometheus-operator</code>, <code>prometheus-server</code>, <code>custom-metrics-api-server</code> and <code>kube-state-metrics</code> attached to a node with <code>system</code> label.</p>

<p>Frontend and backend pod attached to nodes with <code>frontend</code> and <code>backend</code> labels respectively (single pod to a single node), see <a href="https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#inter-pod-affinity-and-anti-affinity-beta-feature" rel="noreferrer">podantiaffinity</a>. </p>

<p>After autoscaling scales <code>backend</code> or <code>frontend</code> pods down, them nodes remains to stay, as there appear to be pods from <code>kube-system</code> namespace, i.e <code>heapster</code>. This leads to a situation when node with <code>frontend</code> / <code>backend</code> label stays alive after downscaling even there's no backend or frontend pod left on it. </p>

<p>The question is: how can I avoid creating <code>kube-system</code> pods on the nodes, that serving my application (if this is really sane and possible)? </p>

<p>Guess, I should use taints and tolerations for <code>backend</code> and <code>frontend</code> nodes, but how it can be combined with HPA and in-cluster node autoscaler?</p>

<p>Seems like <a href="https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/" rel="noreferrer">taints and tolerations</a> did the trick. </p>

<p>Create a cluster with a default node pool (for monitoring and <code>kube-system</code>):</p>

<pre><code>gcloud container --project "my-project-id" clusters create "app-europe" \
  --zone "europe-west1-b" --username="admin" --cluster-version "1.9.4-gke.1" --machine-type "custom-2-4096" \
  --image-type "COS" --disk-size "10" --num-nodes "1" --network "default" --enable-cloud-logging --enable-cloud-monitoring \
  --maintenance-window "01:00" --node-labels=region=europe-west1,role=system
</code></pre>

<p>Create node pool for your application:</p>

<pre><code>gcloud container --project "my-project-id" node-pools create "frontend" \
      --cluster "app-europe" --zone "europe-west1-b" --machine-type "custom-2-2048" --image-type "COS" \
      --disk-size "10" --node-labels=region=europe-west1,role=frontend \
      --node-taints app=frontend:NoSchedule \
      --enable-autoscaling --num-nodes "1" --min-nodes="1" --max-nodes="3"
</code></pre>

<p>then add <code>nodeAffinity</code> and <code>tolerations</code> sections to a pods template <code>spec</code> in your deployment manifest:</p>

<pre><code>  tolerations:
  - key: "app"
    operator: "Equal"
    value: "frontend"
    effect: "NoSchedule"
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: beta.kubernetes.io/instance-type
            operator: In
            values:
            - custom-2-2048
        - matchExpressions:
          - key: role
            operator: In
            values:
            - frontend
</code></pre>
