<p>I have an existing cluster running k8s version <code>1.12.8</code> on AWS EC2. The cluster contains several pods - some serving web traffic and others configured as scheduled CronJobs. The cluster has been running fine in it's current configuration for at least 6 months, with CronJobs running every 5 minutes.</p>
<p>Recently, the CronJobs simply stopped. Viewing the pods via <code>kubectl</code> shows all the scheduled CronJobs last run was at roughly the same time. Logs sent to AWS Cloudwatch show no error output, and stop at the same time <code>kubectl</code> shows for the last run.</p>
<p>In trying to diagnose this issue I have found a broader pattern of the cluster being unresponsive to changes, eg: I cannot retrieve logs or nodes via kubectl.</p>
<p>I deleted Pods in Replica Sets and they never return. I've set autoscale values on Replica Sets and nothing happens.</p>
<p>Investigation of the <code>kubelet</code> logs on the master instance revealed repeating errors, coinciding with the time the failure was first noticed:</p>
<pre><code>I0805 03:17:54.597295 2730 kubelet.go:1928] SyncLoop (PLEG): &quot;kube-scheduler-ip-x-x-x-x.z-west-y.compute.internal_kube-system(181xxyyzz)&quot;, event: &amp;pleg.PodLifecycleEvent{ID:&quot;181xxyyzz&quot;, Type:&quot;ContainerDied&quot;, Data:&quot;405ayyzzz&quot;}
        ...
E0805 03:18:10.867737 2730 kubelet_node_status.go:378] Error updating node status, will retry: failed to patch status &quot;{\&quot;status\&quot;:{\&quot;$setElementOrder/conditions\&quot;:[{\&quot;type\&quot;:\&quot;NetworkUnavailable\&quot;},{\&quot;type\&quot;:\&quot;OutOfDisk\&quot;},{\&quot;type\&quot;:\&quot;MemoryPressure\&quot;},{\&quot;type\&quot;:\&quot;DiskPressure\&quot;},{\&quot;type\&quot;:\&quot;PIDPressure\&quot;},{\&quot;type\&quot;:\&quot;Ready\&quot;}],&quot;conditions\&quot;:[{\&quot;lastHeartbeatTime\&quot;:\&quot;2020-08-05T03:18:00Z\&quot;,\&quot;type\&quot;:\&quot;OutOfDisk\&quot;},{\&quot;lastHeartbeatTime\&quot;:\&quot;2020-08-05T03:18:00Z\&quot;,\&quot;type\&quot;:\&quot;MemoryPressure\&quot;},{\&quot;lastHeartbeatTime\&quot;:\&quot;2020-08-05T03:18:00Z\&quot;,\&quot;type\&quot;:\&quot;DiskPressure\&quot;},{\&quot;lastHeartbeatTime\&quot;:\&quot;2020-08-05T03:18:00Z\&quot;,\&quot;type\&quot;:\&quot;PIDPressure\&quot;},{\&quot;lastHeartbeatTime\&quot;:\&quot;2020-08-05T03:18:00Z\&quot;,\&quot;type\&quot;:\&quot;Ready\&quot;}]}}&quot; for node &quot;ip-172-20-60-88.eu-west-2.compute.internal&quot;: Patch https://127.0.0.1/api/v1/nodes/ip-x-x-x-x.z-west-y.compute.internal/status?timeout=10s: context deadline exceeded (Client.Timeout exceeded while awaiting headers)
...
E0805 03:18:20.869436 2730 kubelet_node_status.go:378] Error updating node status, will retry: error getting node &quot;ip-172-20-60-88.eu-west-2.compute.internal&quot;: Get https://127.0.0.1/api/v1/nodes/ip-172-20-60-88.eu-west-2.compute.internal?timeout=10s: context deadline exceeded (Client.Timeout exceeded while awaiting headers)
</code></pre>
<p>Running <code>docker ps</code> on the master node shows that both <code>k8s_kube-controller-manager_kube-controller-manager</code> and <code>k8s_kube-scheduler_kube-scheduler</code> containers were started 6 days ago, where the other k8s containers are at 8+ months.</p>
<p><strong>tl;dr</strong>
A container on my main node (likely <code>kube-scheduler</code>, <code>kube-controller-manager</code> or both) died. The containers have come back up but are unable to communicate with the existing nodes - this is preventing any scheduled CronJobs or new deployments from being satisfied.</p>
<p>How can re-configure kubelet and associated services on the master node to communicate again with the worker nodes?</p>

<p>From the docs on <a href="https://kubernetes.io/docs/tasks/debug-application-cluster/debug-cluster/" rel="nofollow noreferrer">Troubleshoot Clusters</a></p>
<p>Digging deeper into the cluster requires logging into the relevant machines. Here are the locations of the relevant log files. (note that on systemd-based systems, you may need to use <code>journalctl</code> instead)</p>
<p><strong>Master Nodes</strong></p>
<p><code>/var/log/kube-apiserver.log</code> - API Server, responsible for serving the API</p>
<p><code>/var/log/kube-scheduler.log</code> - Scheduler, responsible for making scheduling decisions</p>
<p><code>/var/log/kube-controller-manager.log</code> - Controller that manages replication controllers</p>
<p><strong>Worker Nodes</strong></p>
<p><code>/var/log/kubelet.log</code> - Kubelet, responsible for running containers on the node
<code>/var/log/kube-proxy.log</code> - Kube Proxy, responsible for service load balancing</p>
<p>Another way to get logs is to use <code>docker ps</code> to get <code>containerid</code> and then use <code>docker logs containerid</code></p>
<p>If you have (which you should) a monitoring system setup using prometheus and Grafana you can check metrics such as high cpu load on API Server pods</p>
