<p>I had a working 3 node k8s cluster (v1.21.0 on Ubuntu 20.04 bare metal) installed using <code>kubeadm</code>. I removed one of the nodes and re-installed it from scratch (wipe disks, new OS but IP address is the same). Now it is unable to join the cluster:</p>
<pre><code># kubeadm join k8s.example.com:6443 --token who21h.jolq7z79twv7bf4m \
--discovery-token-ca-cert-hash sha256:f63c5786cea2be46c999f4b5c595abd0aa24896c3b37616c347df318d7406c00 \
--control-plane
...
[check-etcd] Checking that the etcd cluster is healthy
error execution phase check-etcd: etcd cluster is not healthy: failed to dial endpoint https://65.21.128.36:2379 with maintenance client: context deadline exceeded
To see the stack trace of this error execute with --v=5 or higher
</code></pre>
<p>I ran the same (after <code>kubeadm reset</code>) with <code>--v=5</code> and it gets stuck logging these:</p>
<pre><code>Failed to get etcd status for https://123.123.123.123:2379: failed to dial endpoint https://123.123.123.123:2379 with maintenance client: context deadline exceeded
</code></pre>
<p><code>123.123.123.123</code> is the IP address for the node I am trying to return to the cluster.</p>
<p>Running <code>kubectl get nodes</code> on one of the other masters just lists the 2 remaining masters. I removed the node in question properly:</p>
<pre><code>kubectl get nodes
kubectl drain &lt;node-name&gt;
kubectl drain &lt;node-name&gt; --ignore-daemonsets --delete-local-data
kubectl delete node &lt;node-name&gt;
</code></pre>
<p>Any ideas? Tx.</p>

<p>Take a closer look at the error message you get:</p>
<pre><code>Failed to get etcd status for https://123.123.123.123:2379: failed to dial endpoint https://123.123.123.123:2379 with maintenance client: context deadline exceeded
</code></pre>
<p>This is quite common issue, related with <code>etcd</code> cluster, which is well-documented. Compare with the following threads:</p>
<ul>
<li><a href="https://github.com/kubernetes/kubernetes/issues/81071" rel="noreferrer">Control plain won't join #81071</a></li>
<li><a href="https://github.com/kubernetes/kubeadm/issues/1432" rel="noreferrer">kubeadm join is not fault tolerant to etcd endpoint failures #1432</a></li>
<li><a href="https://superuser.com/questions/1473905/etcd-becomes-unhealthy-after-i-delete-one-of-master-node-i-am-look-for-a-fix">etcd becomes unhealthy after I delete one of master node, I am look for a fix</a></li>
<li><a href="https://docs.openshift.com/container-platform/3.10/admin_guide/assembly_restore-etcd-quorum.html" rel="noreferrer">Restoring etcd quorum</a></li>
</ul>
<p>Specifically, this is related with the loss of <code>etcd</code> quorum. You can check it as described <a href="https://docs.openshift.com/container-platform/3.10/admin_guide/assembly_restore-etcd-quorum.html" rel="noreferrer">here</a>.</p>
<p>The solution is described step by step <a href="https://github.com/kubernetes/kubernetes/issues/81071#issuecomment-519143803" rel="noreferrer">in this comment</a>:</p>
<blockquote>
<p>For the record here the command to run on one of the remaining etcd
pod :</p>
<p>Find the id of the member to remove</p>
<pre><code>ETCDCTL_API=3 etcdctl --endpoints 127.0.0.1:2379 --cacert /etc/kubernetes/pki/etcd/ca.crt --cert
/etc/kubernetes/pki/etcd/server.crt --key
/etc/kubernetes/pki/etcd/server.key member list
5a4945140f0b39d9, started, sbg2-k8s001, https://192.168.208.12:2380, https://192.168.208.12:2379
740381e3c57ef823, started, gra3-k8s001, https://192.168.208.13:2380, https://192.168.208.13:2379
77a8fbb530b10f4a, started, rbx4-k8s001, https://192.168.208.14:2380, https://192.168.208.14:2379
</code></pre>
<p>I want to remove 740381e3c57ef823</p>
<pre><code>ETCDCTL_API=3 etcdctl --endpoints 127.0.0.1:2379 --cacert /etc/kubernetes/pki/etcd/ca.crt --cert
/etc/kubernetes/pki/etcd/server.crt --key
/etc/kubernetes/pki/etcd/server.key member remove 740381e3c57ef823
Member 740381e3c57ef823 removed from cluster a2c90ef66bb95cc9
</code></pre>
<p>Checking</p>
<pre><code>ETCDCTL_API=3 etcdctl --endpoints 127.0.0.1:2379 --cacert /etc/kubernetes/pki/etcd/ca.crt --cert
/etc/kubernetes/pki/etcd/server.crt --key
/etc/kubernetes/pki/etcd/server.key member list
5a4945140f0b39d9, started, sbg2-k8s001, https://192.168.208.12:2380, https://192.168.208.12:2379
77a8fbb530b10f4a, started, rbx4-k8s001, https://192.168.208.14:2380, https://192.168.208.14:2379
</code></pre>
<p>Now I can join my new master.</p>
</blockquote>
