<p>I have multimaster kubernetes cluster setup using Kubespray.
I ran an application using helm on it which increased the load on master drastically. this made master almost inaccessible. After that I shutdown masters one by one and increased RAM and CPU on them. But after rebooting, both apiserver and scheduler pods are failing to start. They are in "CreateContainerError" state.</p>

<p>APIserver is logging lot of errors with the message <code>x509: certificate has expired or is not yet valid</code>.</p>

<p>There are other threads for this error and most of them suggest to fix apiserver or cluster certificates. But this is newly setup cluster and certificates are valid till 2020.</p>

<p>Here are some details of my cluster.</p>

<pre><code>CentOS Linux release: 7.6.1810 (Core)
Docker version: 18.06.1-ce, build e68fc7a
</code></pre>

<p>Kubernetes Version </p>

<pre><code>Client Version: v1.13.2
Server Version: v1.13.2
</code></pre>

SunilThorat
<p>It is very much possible that during shutdown/reboot, docker containers for apiserver and scheduler got exited with non zero exit status like <strong>255</strong>. I sugggest you to first remove all containers with non zero exit status using docker rm command. Do this on all masters, rather on worker nodes as well.</p>

<p>By default, kubernetes starts new pods for all services (apiserver, shceduler, controller-manager, dns, pod network etc.) after a reboot. You can see newly started containers for these services using docker commands
 example:</p>

<pre><code>docker ps -a | grep "kube-apiserver" OR
docker ps -a | grep "kube-scheduler"
</code></pre>

<p>after removing exited containers, I believe, new pods for apiserver and scheduler should run properly in the cluster and should be in "Running" status.</p>
