<p>I have a problem with controller-manager and scheduler not responding, that is not related to github issues I've found (<a href="https://github.com/rancher/rancher/issues/11496" rel="nofollow noreferrer">rancher#11496</a>, <a href="https://github.com/Azure/AKS/issues/173" rel="nofollow noreferrer">azure#173</a>, …)</p>

<p>Two days ago we had a memory overflow by one POD on one Node in our 3-node HA cluster. After that rancher webapp was not accessible, we found the compromised pod and scaled it to 0 over kubectl. But that took some time, figuring everything out.</p>

<p>Since then rancher webapp is working properly, but there are continuous alerts from controller-manager and scheduler not working. Alerts are not consist, sometimes they are both working, some times their health check urls are refusing connection.</p>

<pre><code>NAME                 STATUS      MESSAGE                                                                                     ERROR
controller-manager   Unhealthy   Get http://127.0.0.1:10252/healthz: dial tcp 127.0.0.1:10252: connect: connection refused
scheduler            Healthy     ok                                                                                     
etcd-0               Healthy     {"health": "true"}                                                                     
etcd-2               Healthy     {"health": "true"}                                                                     
etcd-1               Healthy     {"health": "true"}
</code></pre>

<p>Restarting controller-manager and scheduler on compromised Node hasn’t been effective. Even reloading all of the components with</p>

<p><code>docker restart kube-apiserver kubelet kube-controller-manager kube-scheduler kube-proxy</code> 
wasn’t effective either.</p>

<p><strong>Can someone please help me figure out the steps towards troubleshooting and fixing this issue without downtime on running containers?</strong></p>

<p>Nodes are hosted on DigitalOcean on servers with 4 Cores and 8GB of RAM each (Ubuntu 16, Docker 17.03.3).</p>

<p>Thanks in advance !</p>

<p>The first area to look at would be your logs... Can you export the following logs and attach them?</p>

<pre><code>/var/log/kube-controller-manager.log
</code></pre>

<p>The controller manager is an endpoint, so you will need to do a "get endpoint".  Can you run the following:</p>

<pre><code>kubectl -n kube-system get endpoints kube-controller-manager
</code></pre>

<p>and</p>

<pre><code>kubectl -n kube-system describe endpoints kube-controller-manager
</code></pre>

<p>and</p>

<pre><code>kubectl -n kube-system get endpoints kube-controller-manager -o jsonpath='{.metadata.annotations.control-plane\.alpha\.kubernetes\.io/leader}'
</code></pre>
