<h1>Actual Issue:</h1>

<p>Unable to start kubernetes API, due to which, unable to intite kube services like:
kubectl version
kubect get nodes</p>

<pre><code>/home/ubuntu# kubectl version
Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.0", GitCommit:"641856db18352033a0d96dbc99153fa3b27298e5", GitTreeState:"clean", BuildDate:"2019-03-25T15:53:57Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"}



   The connection to the server localhost:8080 was refused - did you specify the right host or port  ?
</code></pre>

<h1>Background of the issue:</h1>

<p>Docker is installed.
Using below, kube components are installed:</p>

<pre><code>apt-get update &amp;&amp; apt-get install -y kubeadm kubelet kubectl
</code></pre>

<p>But, when executing <code>kubeadm init --apiserver-advertise-address=$myip --ignore-preflight-errors=all:</code></p>

<pre><code>I0408 09:09:07.316109       1 client.go:352] scheme "" not registered, fallback to default scheme
I0408 09:09:07.319904       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{127.0.0.1:2379 0  &lt;nil&gt;}]
I0408 09:09:07.323010       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{127.0.0.1:2379 &lt;nil&gt;}]
W0408 09:09:07.332669       1 clientconn.go:1251] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 0  &lt;nil&gt;}. Err :connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused". Reconnecting...
I0408 09:09:08.325625       1 client.go:352] parsed scheme: ""
I0408 09:09:08.325650       1 client.go:352] scheme "" not registered, fallback to default scheme
I0408 09:09:08.325707       1 asm_amd64.s:1337] ccResolverWrapper: sending new addresses to cc: [{127.0.0.1:2379 0  &lt;nil&gt;}]
I0408 09:09:08.325768       1 asm_amd64.s:1337] balancerWrapper: got update addr from Notify: [{127.0.0.1:2379 &lt;nil&gt;}]
W0408 09:09:08.326158       1 clientconn.go:1251] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 0  &lt;nil&gt;}. Err :connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused". Reconnecting...
</code></pre>

<p>Getting the above in the kube api container logs. This is a fresh install .Also tried </p>

<pre><code>sudo cp /etc/kubernetes/admin.conf $HOME/
sudo chown $(id -u):$(id -g) $HOME/admin.conf
export KUBECONFIG=$HOME/admin.conf
</code></pre>

<h1>Expected Results:</h1>

<p>kubectl version: should give only the version details without any connection issue message
Kubectl get nodes: should give the details of Master node and status</p>

<p>I spent a couple of hours on that.<br />
In my case, I was using Ubuntu 22.04 and Kubernetes 1.24.</p>
<p>API was restarting all the time, I didn't find something in kubelet logs:</p>
<pre><code>service kubelet status 
journalctl -xeu kubelet
</code></pre>
<p>I checked the API logs through:</p>
<pre><code>/var/log/containers/kube-apiserver-XXXX
</code></pre>
<p>I saw the same error:</p>
<blockquote>
<p>grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379</p>
</blockquote>
<p><strong>Solution:</strong></p>
<pre><code>containerd config default | tee /etc/containerd/config.toml
sed -i 's/SystemdCgroup = false/SystemdCgroup = true/g' /etc/containerd/config.toml  
service containerd restart
service kubelet restart  
</code></pre>
<p><strong>Explanation:</strong><br />
The problem is <code>cgroupv2</code> introduced with Ubuntu 21.04 and above(Debian since version 11).<br />
You need to tell <code>containerd</code> to use the <code>systemdCgroup</code> driver via the config file(<code>/etc/containerd/config.toml</code>).<br />
I'm using the default config file that containerd provides ( <code>containerd config default | sudo tee /etc/containerd/config.toml</code>).<br />
Don't enable systemd_cgroup inside the &quot;io.containerd.grpc.v1.cri&quot; section, because the plugin doesn't seem support this anymore(status of the service will print following log:<br />
&quot;failed to load plugin io.containerd.grpc.v1.cri&quot; error=&quot;invalid pluginconfig: systemd_cgroup only works for runtime io.containerd.runtime.v1.linux&quot;)<br />
You need to enable the SystemdCgroup(=true) flag inside the section:</p>
<pre><code>[plugins.&quot;io.containerd.grpc.v1.cri&quot;.containerd.runtimes.runc.options]&quot;
</code></pre>
<p>Restart the <code>containerd-service</code> and then the <code>kubelet-service</code> or reboot your machine and then it should work as expected.</p>
<p>Credit: Dennis from slack</p>
