title: GKE kubernetes kube-system resources nodeAffinity
body: <p>I have a multi-regional testing setup on GKE k8s 1.9.4. Every cluster has:</p>
  
  <ul>
  <li>an ingress, configured with <code>kubemci</code></li>
  <li>3 node pools with different node labels:
  
  <ul>
  <li>default-pool <code>system</code> (1vCPU / 2GB RAM)</li>
  <li>frontend-pool <code>frontend</code> (2vCPU / 2GB RAM)</li>
  <li>backend-pool <code>backend</code> (1vCPU / 600Mb RAM)</li>
  </ul></li>
  <li>HPA with scaling by the custom metric</li>
  </ul>
  
  <p>So stuff like <code>prometheus-operator</code>, <code>prometheus-server</code>, <code>custom-metrics-api-server</code> and <code>kube-state-metrics</code> attached to a node with <code>system</code> label.</p>
  
  <p>Frontend and backend pod attached to nodes with <code>frontend</code> and <code>backend</code> labels respectively (single pod to a single node), see <a href="https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#inter-pod-affinity-and-anti-affinity-beta-feature" rel="noreferrer">podantiaffinity</a>. </p>
  
  <p>After autoscaling scales <code>backend</code> or <code>frontend</code> pods down, them nodes remains to stay, as there appear to be pods from <code>kube-system</code> namespace, i.e <code>heapster</code>. This leads to a situation when node with <code>frontend</code> / <code>backend</code> label stays alive after downscaling even there's no backend or frontend pod left on it. </p>
  
  <p>The question is: how can I avoid creating <code>kube-system</code> pods on the nodes, that serving my application (if this is really sane and possible)? </p>
  
  <p>Guess, I should use taints and tolerations for <code>backend</code> and <code>frontend</code> nodes, but how it can be combined with HPA and in-cluster node autoscaler?</p>