<p>This is sort of strange behavior in our K8 cluster.</p>
<p>When we try to deploy a new version of our applications we get:</p>
<pre><code>Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container &quot;&lt;container-id&gt;&quot; network for pod &quot;application-6647b7cbdb-4tp2v&quot;: networkPlugin cni failed to set up pod &quot;application-6647b7cbdb-4tp2v_default&quot; network: Get &quot;https://[10.233.0.1]:443/api/v1/namespaces/default&quot;: dial tcp 10.233.0.1:443: connect: connection refused
</code></pre>
<p>I used <code>kubectl get cs</code> and found <code>controller</code> and <code>scheduler</code> in <code>Unhealthy</code> state.</p>
<p>As describer <a href="https://github.com/kubernetes/kubernetes/issues/93472" rel="nofollow noreferrer">here</a> updated <code>/etc/kubernetes/manifests/kube-scheduler.yaml</code> and
<code>/etc/kubernetes/manifests/kube-controller-manager.yaml</code> by commenting <code>--port=0</code></p>
<p>When I checked <code>systemctl status kubelet</code> it was working.</p>
<pre><code>Active: active (running) since Mon 2020-10-26 13:18:46 +0530; 1 years 0 months ago
</code></pre>
<p>I had restarted kubelet service and <code>controller</code> and <code>scheduler</code> were shown healthy.</p>
<p>But <code>systemctl status kubelet</code> shows (soon after restart kubelet it showed running state)</p>
<pre><code>Active: activating (auto-restart) (Result: exit-code) since Thu 2021-11-11 10:50:49 +0530; 3s ago&lt;br&gt;
    Docs: https://github.com/GoogleCloudPlatform/kubernetes&lt;br&gt;  Process: 21234 ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET
</code></pre>
<p>Tried adding <code>Environment=&quot;KUBELET_SYSTEM_PODS_ARGS=--pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true --fail-swap-on=false&quot; </code> to <code>/etc/systemd/system/kubelet.service.d/10-kubeadm.conf</code> as described <a href="http://Active:%20activating%20(auto-restart)%20(Result:%20exit-code)%20since%20Thu%202021-11-11%2010:50:49%20+0530;%203s%20ago%20%20%20%20%20%20Docs:%20https://github.com/GoogleCloudPlatform/kubernetes%20%20%20Process:%2021234%20ExecStart=/usr/bin/kubelet%20$KUBELET_KUBECONFIG_ARGS%20$KUBELET_CONFIG_ARGS%20$KUBELET_KUBEADM_ARGS%20$KUBELET" rel="nofollow noreferrer">here</a>, but still its not working properly.</p>
<p>Also removed <code>--port=0</code> comment in above mentioned manifests and tried restarting,still same result.</p>
<p><strong>Edit:</strong> This issue was due to <code>kubelet</code> certificate expired and fixed following <a href="https://github.com/kubernetes/kubeadm/issues/2054#issuecomment-606916146" rel="nofollow noreferrer">these</a> steps. If someone faces this issue, make sure <code>/var/lib/kubelet/pki/kubelet-client-current.pem</code> certificate and key values are base64 encoded when placing on <code>/etc/kubernetes/kubelet.conf</code></p>
<p>Many other suggested <code>kubeadm init</code> again. But this cluster was created using <code>kubespray</code> no manually added nodes.</p>
<p>We have baremetal k8 running on Ubuntu 18.04.
K8: v1.18.8</p>
<p>We would like to know any debugging and fixing suggestions.</p>
<p>PS:<br>
When we try to <code>telnet 10.233.0.1 443</code> from any node, first attempt fails and second attempt success.</p>
<p>Edit: Found this in <code>kubelet</code> service logs</p>
<pre><code>Nov 10 17:35:05 node1 kubelet[1951]: W1110 17:35:05.380982    1951 docker_sandbox.go:402] failed to read pod IP from plugin/docker: networkPlugin cni failed on the status hook for pod &quot;app-7b54557dd4-bzjd9_default&quot;: unexpected command output nsenter: cannot open /proc/12311/ns/net: No such file or directory
</code></pre>

Sachith Muhandiram
<p>Posting comment as the community wiki answer for better visibility</p>
<hr />
<p>This issue was due to <code>kubelet</code> certificate expired and fixed following <a href="https://github.com/kubernetes/kubeadm/issues/2054#issuecomment-606916146" rel="nofollow noreferrer">these steps</a>. If someone faces this issue, make sure <code>/var/lib/kubelet/pki/kubelet-client-current.pem</code> certificate and key values are <code>base64</code> encoded when placing on <code>/etc/kubernetes/kubelet.conf</code></p>
