<p>I am new to k8s and I found a problem that I can not resolve.</p>
<p>I am building a HA cluster of Master nodes. I am running some tests (removing one node and adding the node again). Through this process I noticed that the etcd cluster does not update the cluster list.</p>
<p>Sample of problem below:</p>
<pre class="lang-sh prettyprint-override"><code>$ kubectl get pods -A
NAMESPACE                NAME                                                 READY   STATUS    RESTARTS   AGE
cri-o-metrics-exporter   cri-o-metrics-exporter-77c9cf9746-qlp4d              0/1     Pending   0          16h
haproxy-controller       haproxy-ingress-769d858699-b8r8q                     0/1     Pending   0          16h
haproxy-controller       ingress-default-backend-5fd4986454-kvbw8             0/1     Pending   0          16h
kube-system              calico-kube-controllers-574d679d8c-tkcjj             1/1     Running   3          16h
kube-system              calico-node-95t6l                                    1/1     Running   2          16h
kube-system              calico-node-m5txs                                    1/1     Running   2          16h
kube-system              coredns-7588b55795-gkfjq                             1/1     Running   2          16h
kube-system              coredns-7588b55795-lxpmj                             1/1     Running   2          16h
kube-system              etcd-masterNode1                                     1/1     Running   2          16h
kube-system              etcd-masterNode2                                     1/1     Running   2          16h
kube-system              kube-apiserver-masterNode1                           1/1     Running   3          16h
kube-system              kube-apiserver-masterNode2                           1/1     Running   3          16h
kube-system              kube-controller-manager-masterNode1                  1/1     Running   4          16h
kube-system              kube-controller-manager-masterNode2                  1/1     Running   4          16h
kube-system              kube-proxy-5q6xs                                     1/1     Running   2          16h
kube-system              kube-proxy-k8p6h                                     1/1     Running   2          16h
kube-system              kube-scheduler-masterNode1                           1/1     Running   3          16h
kube-system              kube-scheduler-masterNode2                           1/1     Running   6          16h
kube-system              metrics-server-575bd7f776-jtfsh                      0/1     Pending   0          16h
kubernetes-dashboard     dashboard-metrics-scraper-6f78bc588b-khjjr           1/1     Running   2          16h
kubernetes-dashboard     kubernetes-dashboard-978555c5b-9jsxb                 1/1     Running   2          16h
$ kubectl exec etcd-masterNode2 -n kube-system -it -- sh
sh-5.0# etcdctl --cacert /etc/kubernetes/pki/etcd/ca.crt --cert /etc/kubernetes/pki/etcd/peer.crt --key /etc/kubernetes/pki/etcd/peer.key member list -w table
+------------------+---------+----------------------------+---------------------------+---------------------------+------------+
|        ID        | STATUS  |            NAME            |        PEER ADDRS         |       CLIENT ADDRS        | IS LEARNER |
+------------------+---------+----------------------------+---------------------------+---------------------------+------------+
| 4c209e5bc1ca9593 | started |         masterNode1        |     https://IP1:2380      |     https://IP1:2379      |      false |
| 676d4bfab319fa22 | started |         masterNode2        |     https://IP2:2380      |     https://IP2:2379      |      false |
| a9af4b00e33f87d4 | started |         masterNode3        |     https://IP3:2380      |     https://IP3:2379      |      false |
+------------------+---------+----------------------------+---------------------------+---------------------------+------------+
sh-5.0# exit
$ kubectl get nodes
NAME                         STATUS   ROLES    AGE   VERSION
masterNode1                  Ready    master   16h   v1.19.0
masterNode2                  Ready    master   16h   v1.19.0
</code></pre>
<p>I assume that I am removing correctly the node from the cluster. The procedure that I am following:</p>
<ol>
<li>kubectl drain --ignore-daemonsets --delete-local-data</li>
<li>kubectl delete</li>
<li>node kubeadm reset</li>
<li>rm -f /etc/cni/net.d/* # Removing CNI configuration</li>
<li>rm -rf /var/lib/kubelet # Removing /var/lib/kubeler dir</li>
<li>rm -rf /var/lib/etcd # Removing /var/lib/etcd</li>
<li>iptables -F &amp;&amp; iptables -t nat -F &amp;&amp; iptables -t mangle -F &amp;&amp; iptables -X &amp;&amp; iptables -t filter -F &amp;&amp; iptables -t filter -X # Removing iptables</li>
<li>ipvsadm --clear</li>
<li>rm -rf /etc/kubernetes # Removing /etc/kubernetes (in case of character change)</li>
</ol>
<p>I am running kubernetes with version <code>1.19.0</code> and etcd <code>etcd:3.4.9-1</code>.</p>
<p>The cluster is running on bare metal nodes.</p>
<p>Is this a bug or I am not removing the node correctly from the etcd cluster?</p>

<p>Thanks to <a href="https://stackoverflow.com/users/12237732/mariusz-k">Mariusz K.</a> I found the answer to my problem. In case that someone else might have the same problem here is how I solved it.</p>
<p>First query the cluster (HA) for the etcd members (sample of code):</p>
<pre class="lang-sh prettyprint-override"><code>$ kubectl exec etcd-&lt; nodeNameMasterNode &gt; -n kube-system -- etcdctl --cacert /etc/kubernetes/pki/etcd/ca.crt --cert /etc/kubernetes/pki/etcd/peer.crt --key /etc/kubernetes/pki/etcd/peer.key member list
1863b58e85c8a808, started, nodeNameMaster1, https://IP1:2380, https://IP1:2379, false
676d4bfab319fa22, started, nodeNameMaster2, https://IP2:2380, https://IP2:2379, false
b0c50c50d563ed51, started, nodeNameMaster3, https://IP3:2380, https://IP3:2379, false
</code></pre>
<p>Then once you have the list of nodes you can remove any member you want. Sample of code:</p>
<pre class="lang-sh prettyprint-override"><code>kubectl exec etcd-nodeNameMaster1 -n kube-system -- etcdctl --cacert /etc/kubernetes/pki/etcd/ca.crt --cert /etc/kubernetes/pki/etcd/peer.crt --key /etc/kubernetes/pki/etcd/peer.key member remove b0c50c50d563ed51
Member b0c50c50d563ed51 removed from cluster d1e1de99e3d19634
</code></pre>
<p>I wanted to be able to remove a member from the etcd cluster without the need to connect to the pod and run a secondary command. This way I execute the command to the pod through exec.</p>
