<p>I have configured 1 master 2 workers.
after installation successfully kubernetes. It is OK with worker1 joining cluster but I can not join worker2 to the cluster
because kubelet service is not running. <strong>It seems like the kubelet isn't running or healthy</strong></p>
<p><strong>sudo kubectl get nodes:</strong></p>
<blockquote>
<p>NAME      STATUS     ROLES                  AGE     VERSION<br />
master1   Ready      control-plane,master   23m     v1.22.2<br />
node1     NotReady                    4m13s   v1.22.2</p>
</blockquote>
<p>I want to know why the kubelet service is not running.</p>
<p><strong>Here kubelet logs.</strong></p>
<pre><code>The start-up result is RESULT.
Dec 04 20:21:26 node2 kubelet[25435]: Flag --network-plugin has been deprecated, will be removed along with dockershim.
Dec 04 20:21:26 node2 kubelet[25435]: Flag --network-plugin has been deprecated, will be removed along with dockershim.
Dec 04 20:21:26 node2 kubelet[25435]: I1204 20:21:26.659131   25435 server.go:440] &quot;Kubelet version&quot; kubeletVersion=&quot;v1.22.2&quot;
Dec 04 20:21:26 node2 kubelet[25435]: I1204 20:21:26.659587   25435 server.go:868] &quot;Client rotation is on, will bootstrap in background&quot;
Dec 04 20:21:26 node2 kubelet[25435]: I1204 20:21:26.678863   25435 certificate_store.go:130] Loading cert/key pair from &quot;/var/lib/kubelet/pki/kubelet-client-current.pem&quot;.
Dec 04 20:21:26 node2 kubelet[25435]: I1204 20:21:26.684321   25435 dynamic_cafile_content.go:155] &quot;Starting controller&quot; name=&quot;client-ca-bundle::/etc/kubernetes/pki/ca.crt&quot;
Dec 04 20:21:26 node2 kubelet[25435]: I1204 20:21:26.728096   25435 server.go:687] &quot;--cgroups-per-qos enabled, but --cgroup-root was not specified.  defaulting to /&quot;
Dec 04 20:21:26 node2 kubelet[25435]: I1204 20:21:26.728320   25435 container_manager_linux.go:280] &quot;Container manager verified user specified cgroup-root exists&quot; cgroupRoot=[]
Dec 04 20:21:26 node2 kubelet[25435]: I1204 20:21:26.728388   25435 container_manager_linux.go:285] &quot;Creating Container Manager object based on Node Config&quot; nodeConfig={RuntimeCgroupsName:
Dec 04 20:21:26 node2 kubelet[25435]: I1204 20:21:26.729329   25435 topology_manager.go:133] &quot;Creating topology manager with policy per scope&quot; topologyPolicyName=&quot;none&quot; topologyScopeName=&quot;c
Dec 04 20:21:26 node2 kubelet[25435]: I1204 20:21:26.729345   25435 container_manager_linux.go:320] &quot;Creating device plugin manager&quot; devicePluginEnabled=true
Dec 04 20:21:26 node2 kubelet[25435]: I1204 20:21:26.729367   25435 state_mem.go:36] &quot;Initialized new in-memory state store&quot;
Dec 04 20:21:26 node2 kubelet[25435]: I1204 20:21:26.729408   25435 kubelet.go:314] &quot;Using dockershim is deprecated, please consider using a full-fledged CRI implementation&quot;
Dec 04 20:21:26 node2 kubelet[25435]: I1204 20:21:26.729430   25435 client.go:78] &quot;Connecting to docker on the dockerEndpoint&quot; endpoint=&quot;unix:///var/run/docker.sock&quot;
Dec 04 20:21:26 node2 kubelet[25435]: I1204 20:21:26.729441   25435 client.go:97] &quot;Start docker client with request timeout&quot; timeout=&quot;2m0s&quot;
Dec 04 20:21:26 node2 kubelet[25435]: I1204 20:21:26.744324   25435 docker_service.go:566] &quot;Hairpin mode is set but kubenet is not enabled, falling back to HairpinVeth&quot; hairpinMode=promiscu
Dec 04 20:21:26 node2 kubelet[25435]: I1204 20:21:26.744354   25435 docker_service.go:242] &quot;Hairpin mode is set&quot; hairpinMode=hairpin-veth
Dec 04 20:21:26 node2 kubelet[25435]: I1204 20:21:26.744554   25435 cni.go:239] &quot;Unable to update cni config&quot; err=&quot;no networks found in /etc/cni/net.d&quot;
Dec 04 20:21:26 node2 kubelet[25435]: I1204 20:21:26.750011   25435 cni.go:239] &quot;Unable to update cni config&quot; err=&quot;no networks found in /etc/cni/net.d&quot;
Dec 04 20:21:26 node2 kubelet[25435]: I1204 20:21:26.750260   25435 docker_service.go:257] &quot;Docker cri networking managed by the network plugin&quot; networkPluginName=&quot;cni&quot;
Dec 04 20:21:26 node2 kubelet[25435]: I1204 20:21:26.753050   25435 cni.go:239] &quot;Unable to update cni config&quot; err=&quot;no networks found in /etc/cni/net.d&quot;
Dec 04 20:21:26 node2 kubelet[25435]: I1204 20:21:26.764080   25435 docker_service.go:264] &quot;Docker Info&quot; dockerInfo=&amp;{ID:4UUR:AFJU:SXYE:5IRP:6G6B:SFDY:H3AA:D5ZB:JSDO:GXVQ:UYNG:POJY Containe
Dec 04 20:21:26 node2 kubelet[25435]: E1204 20:21:26.765777   25435 server.go:294] &quot;Failed to run kubelet&quot; err=&quot;failed to run Kubelet: misconfiguration: kubelet cgroup driver: \&quot;systemd\&quot; i
Dec 04 20:21:26 node2 systemd[1]: kubelet.service: Main process exited, code=exited, status=1/FAILURE
Dec 04 20:21:26 node2 systemd[1]: kubelet.service: Failed with result 'exit-code'.

</code></pre>
<p><strong>kubeadm join logs:</strong></p>
<pre><code>I1204 20:27:56.222794   29796 join.go:405] [preflight] found NodeName empty; using OS hostname as NodeName
I1204 20:27:56.223032   29796 initconfiguration.go:116] detected and using CRI socket: /var/run/dockershim.sock
[preflight] Running pre-flight checks
I1204 20:27:56.223834   29796 preflight.go:92] [preflight] Running general checks
I1204 20:27:56.225983   29796 checks.go:245] validating the existence and emptiness of directory /etc/kubernetes/manifests
I1204 20:27:56.226133   29796 checks.go:282] validating the existence of file /etc/kubernetes/kubelet.conf
I1204 20:27:56.226271   29796 checks.go:282] validating the existence of file /etc/kubernetes/bootstrap-kubelet.conf
I1204 20:27:56.226408   29796 checks.go:106] validating the container runtime
I1204 20:27:56.282374   29796 checks.go:132] validating if the &quot;docker&quot; service is enabled and active
I1204 20:27:56.300100   29796 checks.go:331] validating the contents of file /proc/sys/net/bridge/bridge-nf-call-iptables
I1204 20:27:56.300279   29796 checks.go:331] validating the contents of file /proc/sys/net/ipv4/ip_forward
I1204 20:27:56.300580   29796 checks.go:649] validating whether swap is enabled or not
I1204 20:27:56.300738   29796 checks.go:372] validating the presence of executable conntrack
I1204 20:27:56.301009   29796 checks.go:372] validating the presence of executable ip
I1204 20:27:56.301613   29796 checks.go:372] validating the presence of executable iptables
I1204 20:27:56.301801   29796 checks.go:372] validating the presence of executable mount
I1204 20:27:56.302057   29796 checks.go:372] validating the presence of executable nsenter
I1204 20:27:56.302384   29796 checks.go:372] validating the presence of executable ebtables
I1204 20:27:56.302473   29796 checks.go:372] validating the presence of executable ethtool
I1204 20:27:56.302569   29796 checks.go:372] validating the presence of executable socat
I1204 20:27:56.302610   29796 checks.go:372] validating the presence of executable tc
I1204 20:27:56.303072   29796 checks.go:372] validating the presence of executable touch
I1204 20:27:56.303472   29796 checks.go:520] running all checks
I1204 20:27:56.372402   29796 checks.go:403] checking whether the given node name is valid and reachable using net.LookupHost
I1204 20:27:56.373211   29796 checks.go:618] validating kubelet version
I1204 20:27:56.467792   29796 checks.go:132] validating if the &quot;kubelet&quot; service is enabled and active
I1204 20:27:56.485715   29796 checks.go:205] validating availability of port 10250
I1204 20:27:56.486624   29796 checks.go:282] validating the existence of file /etc/kubernetes/pki/ca.crt
I1204 20:27:56.487016   29796 checks.go:432] validating if the connectivity type is via proxy or direct
I1204 20:27:56.487841   29796 join.go:475] [preflight] Discovering cluster-info
I1204 20:27:56.488260   29796 token.go:80] [discovery] Created cluster-info discovery client, requesting info from &quot;192.168.1.53:6443&quot;
I1204 20:27:56.520182   29796 token.go:118] [discovery] Requesting info from &quot;192.168.1.53:6443&quot; again to validate TLS against the pinned public key
I1204 20:27:56.530589   29796 token.go:135] [discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server &quot;192.168.1.53:6443&quot;
I1204 20:27:56.530702   29796 discovery.go:52] [discovery] Using provided TLSBootstrapToken as authentication credentials for the join process
I1204 20:27:56.530924   29796 join.go:489] [preflight] Fetching init configuration
I1204 20:27:56.531171   29796 join.go:534] [preflight] Retrieving KubeConfig objects
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
I1204 20:27:56.549808   29796 interface.go:431] Looking for default routes with IPv4 addresses
I1204 20:27:56.549913   29796 interface.go:436] Default route transits interface &quot;enp0s3&quot;
I1204 20:27:56.550259   29796 interface.go:208] Interface enp0s3 is up
I1204 20:27:56.550564   29796 interface.go:256] Interface &quot;enp0s3&quot; has 2 addresses :[192.168.1.50/24 fe80::a00:27ff:fe7e:db8b/64].
I1204 20:27:56.550644   29796 interface.go:223] Checking addr  192.168.1.50/24.
I1204 20:27:56.550887   29796 interface.go:230] IP found 192.168.1.50
I1204 20:27:56.550955   29796 interface.go:262] Found valid IPv4 address 192.168.1.50 for interface &quot;enp0s3&quot;.
I1204 20:27:56.551237   29796 interface.go:442] Found active IP 192.168.1.50
I1204 20:27:56.563573   29796 preflight.go:103] [preflight] Running configuration dependant checks
I1204 20:27:56.563872   29796 controlplaneprepare.go:219] [download-certs] Skipping certs download
I1204 20:27:56.565399   29796 kubelet.go:112] [kubelet-start] writing bootstrap kubelet config file at /etc/kubernetes/bootstrap-kubelet.conf
I1204 20:27:56.569613   29796 kubelet.go:120] [kubelet-start] writing CA certificate at /etc/kubernetes/pki/ca.crt
I1204 20:27:56.572216   29796 kubelet.go:141] [kubelet-start] Checking for an existing Node in the cluster with name &quot;node2&quot; and status &quot;Ready&quot;
I1204 20:27:56.576685   29796 kubelet.go:155] [kubelet-start] Stopping the kubelet
[kubelet-start] Writing kubelet configuration to file &quot;/var/lib/kubelet/config.yaml&quot;
[kubelet-start] Writing kubelet environment file with flags to file &quot;/var/lib/kubelet/kubeadm-flags.env&quot;
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
I1204 20:28:01.956734   29796 kubelet.go:190] [kubelet-start] preserving the crisocket information for the node
I1204 20:28:01.956911   29796 patchnode.go:31] [patchnode] Uploading the CRI Socket information &quot;/var/run/dockershim.sock&quot; to the Node API object &quot;node2&quot; as an annotation
I1204 20:28:01.957066   29796 cert_rotation.go:137] Starting client certificate rotation controller
[kubelet-check] Initial timeout of 40s passed.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get &quot;http://localhost:10248/healthz&quot;: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get &quot;http://localhost:10248/healthz&quot;: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get &quot;http://localhost:10248/healthz&quot;: dial tcp 127.0.0.1:10248: connect: connection refused.
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get &quot;http://localhost:10248/healthz&quot;: dial tcp 127.0.0.1:10248: connect: connection refused.

</code></pre>

<p>First, check if swap is diabled on your node as you MUST disable swap in order for the kubelet to work properly.</p>
<pre class="lang-sh prettyprint-override"><code>sudo swapoff -a  
sudo sed -i '/ swap / s/^/#/' /etc/fstab
</code></pre>
<p>Also check out if kubernetes and docker cgroup driver is set to same.
From <a href="https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#configuring-a-cgroup-driver" rel="noreferrer">kubernetes documentation</a>:</p>
<blockquote>
<p>Both the container runtime and the kubelet have a property called &quot;cgroup driver&quot;, which is important for the management of cgroups on Linux machines.<br />
Warning:
Matching the container runtime and kubelet cgroup drivers is required or otherwise the kubelet process will fail.</p>
</blockquote>
<p>The <a href="https://kubernetes.io/docs/setup/production-environment/container-runtimes/" rel="noreferrer">Container runtimes</a> page explains that the systemd driver is recommended for kubeadm based setups instead of the cgroupfs driver, because kubeadm manages the kubelet as a systemd service.</p>
<p>For docker:</p>
<pre><code>docker info |grep -i cgroup
</code></pre>
<p>You can add this to <code>/etc/docker/daemon.json</code> to set the docker cgroup driver to systemd:</p>
<pre><code>{
    &quot;exec-opts&quot;: [&quot;native.cgroupdriver=systemd&quot;]
}
</code></pre>
<p>Restart your docker service after making any changes with</p>
<pre><code>sudo systemctl daemon-reload
sudo systemctl restart docker
sudo systemctl restart kubelet
</code></pre>
<p>You can try to execute <code>kubeadm join</code> after performing the above steps.</p>
