<p>I've started working with the docker images and set up Kubernetes. I have fixed everything but I am having problems with the timeout of pod recreations.</p>

<p>If one pod is running on one particular node and if I shut it down, it will take ~5 minutes to recreate the pod on another online node.</p>

<p>I've checked all the possible config files, also set all pod-eviction-timeout, horizontal-pod-autoscaler-downscale, horizontal-pod-autoscaler-downscale-delay flags but it is still not working.</p>

<p>Current kube controller manager config:</p>

<pre><code>spec:
 containers:
 - command:
   - kube-controller-manager
   - --address=192.168.5.135
   - --allocate-node-cidrs=false
   - --authentication-kubeconfig=/etc/kubernetes/controller-manager.conf
   - --authorization-kubeconfig=/etc/kubernetes/controller-manager.conf
   - --client-ca-file=/etc/kubernetes/pki/ca.crt
   - --cluster-cidr=192.168.5.0/24
   - --cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt
   - --cluster-signing-key-file=/etc/kubernetes/pki/ca.key
   - --controllers=*,bootstrapsigner,tokencleaner
   - --kubeconfig=/etc/kubernetes/controller-manager.conf
   - --leader-elect=true
   - --node-cidr-mask-size=24
   - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
   - --root-ca-file=/etc/kubernetes/pki/ca.crt
   - --service-account-private-key-file=/etc/kubernetes/pki/sa.key
   - --use-service-account-credentials=true
   - --horizontal-pod-autoscaler-downscale-delay=20s
   - --horizontal-pod-autoscaler-sync-period=20s
   - --node-monitor-grace-period=40s
   - --node-monitor-period=5s
   - --pod-eviction-timeout=20s
   - --use-service-account-credentials=true
   - --horizontal-pod-autoscaler-downscale-stabilization=20s
image: k8s.gcr.io/kube-controller-manager:v1.13.0
</code></pre>

<p>Thank you.</p>

Jure Potocnik
<p>If <a href="https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/#taint-based-evictions" rel="noreferrer">Taint Based Evictions</a> are present in the pod definition, controller manager will not be able to evict the pod that tolerates the taint. Even if you don't define an eviction policy in your configuration, it gets a default one since <a href="https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#defaulttolerationseconds" rel="noreferrer">Default Toleration Seconds</a> admission controller plugin is enabled by default.</p>

<p>Default Toleration Seconds admission controller plugin configures your pod like below:</p>

<pre><code>tolerations:
- key: node.kubernetes.io/not-ready
  effect: NoExecute
  tolerationSeconds: 300
- key: node.kubernetes.io/unreachable
  operator: Exists
  effect: NoExecute
  tolerationSeconds: 300
</code></pre>

<p>You can verify this by inspecting definition of your pod:</p>

<pre><code>kubectl get pods -o yaml -n &lt;namespace&gt; &lt;pod-name&gt;`
</code></pre>

<p>According to above toleration it takes more than 5 minutes to recreate the pod on another ready node since pod can tolerate <code>not-ready</code> taint for up to 5 minutes. In this case, even if you set <code>--pod-eviction-timeout</code> to 20s, there is nothing controller manager can do because of the tolerations.</p>

<p>But why it takes more than 5 minutes? Because the node will be considered as down after <code>--node-monitor-grace-period</code> which defaults to 40s. After that, pod toleration timer starts.</p>

<hr>

<h3>Recommended Solution</h3>

<p>If you want your cluster to react faster for node outages, you should use taints and tolerations without modifying options. For example, you can define your pod like below:</p>

<pre><code>tolerations:
- key: node.kubernetes.io/not-ready
  effect: NoExecute
  tolerationSeconds: 0
- key: node.kubernetes.io/unreachable
  effect: NoExecute
  tolerationSeconds: 0
</code></pre>

<p>With above toleration your pod will be recreated on a ready node just after the current node marked as not ready. This should take less then a minute since <code>--node-monitor-grace-period</code> is default to 40s.</p>

<h3>Available Options</h3>

<p>If you want to be in control of these timings below you will find plenty of options to do so. However, modifying these options should be avoided. If you use tight timings which might create an overhead on etcd as every node will try to update its status very often.</p>

<p>In addition to this, currently it is not clear how to propagate changes in controller manager, api server and kubelet configuration to all nodes in a living cluster. Please see <a href="https://github.com/kubernetes/kubeadm/issues/970" rel="noreferrer">Tracking issue for changing the cluster</a> and <a href="https://github.com/kubernetes/enhancements/issues/281" rel="noreferrer">Dynamic Kubelet Configuration</a>. As of this writing, <a href="https://kubernetes.io/docs/tasks/administer-cluster/reconfigure-kubelet/" rel="noreferrer">reconfiguring a node's kubelet in a live cluster</a> is in beta.</p>

<p>You can configure control plane and kubelet during kubeadm init or join phase. Please refer to <a href="https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/control-plane-flags/" rel="noreferrer">Customizing control plane configuration with kubeadm</a> and <a href="https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/kubelet-integration/" rel="noreferrer">Configuring each kubelet in your cluster using kubeadm</a> for more details.</p>

<p>Assuming you have a single node cluster:</p>

<ul>
<li><a href="https://kubernetes.io/docs/reference/command-line-tools-reference/kube-controller-manager/" rel="noreferrer">controller manager</a> includes:

<ul>
<li><code>--node-monitor-grace-period</code> default 40s</li>
<li><code>--node-monitor-period</code> default 5s</li>
<li><code>--pod-eviction-timeout</code> default 5m0s</li>
</ul></li>
<li><a href="https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/" rel="noreferrer">api server</a> includes:

<ul>
<li><code>--default-not-ready-toleration-seconds</code> default 300</li>
<li><code>--default-unreachable-toleration-seconds</code> default 300</li>
</ul></li>
<li><a href="https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/" rel="noreferrer">kubelet</a> includes:

<ul>
<li><code>--node-status-update-frequency</code> default 10s</li>
</ul></li>
</ul>

<p>If you set up the cluster with <code>kubeadm</code> you can modify:</p>

<ul>
<li><code>/etc/kubernetes/manifests/kube-controller-manager.yaml</code> for controller manager options.</li>
<li><code>/etc/kubernetes/manifests/kube-apiserver.yaml</code> for api server options.</li>
</ul>

<p><strong>Note:</strong> Modifying these files will reconfigure and restart the respective pod in the node.</p>

<p>In order to modify <code>kubelet</code> config you can add below line:</p>

<pre><code>KUBELET_EXTRA_ARGS="--node-status-update-frequency=10s"
</code></pre>

<p>To <code>/etc/default/kubelet</code> (for DEBs), or <code>/etc/sysconfig/kubelet</code> (for RPMs) and then restart kubelet service:</p>

<pre><code>sudo systemctl daemon-reload &amp;&amp; sudo systemctl restart kubelet
</code></pre>
