title: Adding removed etcd member in Kubernetes master
body: <p>I was following Kelsey Hightower's kubernetes-the-hard-way repo and successfully created a cluster with 3 master nodes and 3 worker nodes. Here are the problems encountered when removing one of the etcd members and then adding it back, also with all the steps used:</p>
  
  <p><strong>3 master nodes:</strong><br>
  10.240.0.10 controller-0<br>
  10.240.0.11 controller-1<br>
  10.240.0.12 controller-2</p>
  
  <p>Step 1:</p>
  
  <pre><code>isaac@controller-0:~$ sudo ETCDCTL_API=3 etcdctl member list   --endpoints=https://127.0.0.1:2379   --cacert=/etc/etcd/ca.pem   --cert=/etc/etcd/kubernetes.pem   --key=/etc/etcd/kubernetes-key.pem
  </code></pre>
  
  <p>Result:</p>
  
  <blockquote>
    <p>b28b52253c9d447e, started, controller-2, <a href="https://10.240.0.12:2380" rel="nofollow noreferrer">https://10.240.0.12:2380</a>, <a href="https://10.240.0.12:2379" rel="nofollow noreferrer">https://10.240.0.12:2379</a><br>
    f98dc20bce6225a0, started, controller-0, <a href="https://10.240.0.10:2380" rel="nofollow noreferrer">https://10.240.0.10:2380</a>, <a href="https://10.240.0.10:2379" rel="nofollow noreferrer">https://10.240.0.10:2379</a><br>
    ffed16798470cab5, started, controller-1, <a href="https://10.240.0.11:2380" rel="nofollow noreferrer">https://10.240.0.11:2380</a>, <a href="https://10.240.0.11:2379" rel="nofollow noreferrer">https://10.240.0.11:2379</a></p>
  </blockquote>
  
  <p>Step 2 (Remove etcd member of controller-2):</p>
  
  <pre><code>isaac@controller-0:~$ sudo ETCDCTL_API=3 etcdctl member remove b28b52253c9d447e   --endpoints=https://127.0.0.1:2379   --cacert=/etc/etcd/ca.pem   --cert=/etc/etcd/kubernetes.pem   --key=/etc/etcd/kubernetes-key.pem
  </code></pre>
  
  <p>Step 3 (Add the member back):</p>
  
  <pre><code>isaac@controller-0:~$ sudo ETCDCTL_API=3 etcdctl member add controller-2 --peer-urls=https://10.240.0.12:2380  --endpoints=https://127.0.0.1:2379   --cacert=/etc/etcd/ca.pem   --cert=/etc/etcd/kubernetes.pem   --key=/etc/etcd/kubernetes-key.pem
  </code></pre>
  
  <p>Result:</p>
  
  <blockquote>
    <p>Member 66d450d03498eb5c added to cluster 3e7cc799faffb625
    ETCD_NAME="controller-2"
    ETCD_INITIAL_CLUSTER="controller-2=<a href="https://10.240.0.12:2380,controller-0=https://10.240.0.10:2380,controller-1=https://10.240.0.11:2380" rel="nofollow noreferrer">https://10.240.0.12:2380,controller-0=https://10.240.0.10:2380,controller-1=https://10.240.0.11:2380</a>"
    ETCD_INITIAL_ADVERTISE_PEER_URLS="https://10.240.0.12:2380"
    ETCD_INITIAL_CLUSTER_STATE="existing"</p>
  </blockquote>
  
  <p>Step 4 (run member list command):</p>
  
  <pre><code>isaac@controller-0:~$ sudo ETCDCTL_API=3 etcdctl member list   --endpoints=https://127.0.0.1:2379   --cacert=/etc/etcd/ca.pem   --cert=/etc/etcd/kubernetes.pem   --key=/etc/etcd/kubernetes-key.pem
  </code></pre>
  
  <p>Result: </p>
  
  <blockquote>
    <p>66d450d03498eb5c, unstarted, , <a href="https://10.240.0.12:2380" rel="nofollow noreferrer">https://10.240.0.12:2380</a>,<br>
    f98dc20bce6225a0, started, controller-0, <a href="https://10.240.0.10:2380" rel="nofollow noreferrer">https://10.240.0.10:2380</a>,
    <a href="https://10.240.0.10:2379" rel="nofollow noreferrer">https://10.240.0.10:2379</a>   ffed16798470cab5, started, controller-1,
    <a href="https://10.240.0.11:2380" rel="nofollow noreferrer">https://10.240.0.11:2380</a>, <a href="https://10.240.0.11:2379" rel="nofollow noreferrer">https://10.240.0.11:2379</a></p>
  </blockquote>
  
  <p>Step 5 (Run the command to start etcd in controller-2): </p>
  
  <pre><code>isaac@controller-2:~$ sudo etcd --name controller-2 --listen-client-urls https://10.240.0.12:2379,http://127.0.0.1:2379 --advertise-client-urls https://10.240.0.12:2379 --listen-peer-urls https://10.240.0.12:
  2380 --initial-advertise-peer-urls https://10.240.0.12:2380 --initial-cluster-state existing --initial-cluster controller-0=http://10.240.0.10:2380,controller-1=http://10.240.0.11:2380,controller-2=http://10.240.0.1
  2:2380 --ca-file /etc/etcd/ca.pem --cert-file /etc/etcd/kubernetes.pem --key-file /etc/etcd/kubernetes-key.pem
  </code></pre>
  
  <p>Result: </p>
  
  <blockquote>
    <p>2019-06-09 13:10:14.958799 I | etcdmain: etcd Version: 3.3.9
    2019-06-09 13:10:14.959022 I | etcdmain: Git SHA: fca8add78
    2019-06-09 13:10:14.959106 I | etcdmain: Go Version: go1.10.3
    2019-06-09 13:10:14.959177 I | etcdmain: Go OS/Arch: linux/amd64
    2019-06-09 13:10:14.959237 I | etcdmain: setting maximum number of CPUs to 1, total number of available CPUs is 1
    2019-06-09 13:10:14.959312 W | etcdmain: no data-dir provided, using default data-dir ./controller-2.etcd
    2019-06-09 13:10:14.959435 N | etcdmain: the server is already initialized as member before, starting as etcd member...
    2019-06-09 13:10:14.959575 C | etcdmain: cannot listen on TLS for 10.240.0.12:2380: KeyFile and CertFile are not presented</p>
  </blockquote>
  
  <p>Clearly, the etcd service did not start as expected, so I do the troubleshooting as below:</p>
  
  <pre><code>isaac@controller-2:~$ sudo systemctl status etcd
  </code></pre>
  
  <p>Result:</p>
  
  <blockquote>
    <p>● etcd.service - etcd    Loaded: loaded
    (/etc/systemd/system/etcd.service; enabled; vendor preset: enabled)<br>
    Active: inactive (dead) since Sun 2019-06-09 13:06:55 UTC; 29min ago
         Docs: <a href="https://github.com/coreos" rel="nofollow noreferrer">https://github.com/coreos</a>   Process: 1876 ExecStart=/usr/local/bin/etcd --name controller-2
    --cert-file=/etc/etcd/kubernetes.pem --key-file=/etc/etcd/kubernetes-key.pem --peer-cert-file=/etc/etcd/kubernetes.pem --peer-key-file=/etc/etcd/kube  Main PID: 1876 (code=exited, status=0/SUCCESS) Jun 09 13:06:55 controller-2 etcd[1876]: stopped
    peer f98dc20bce6225a0 Jun 09 13:06:55 controller-2 etcd[1876]:
    stopping peer ffed16798470cab5... Jun 09 13:06:55 controller-2
    etcd[1876]: stopped streaming with peer ffed16798470cab5 (writer) Jun
    09 13:06:55 controller-2 etcd[1876]: stopped streaming with peer
    ffed16798470cab5 (writer) Jun 09 13:06:55 controller-2 etcd[1876]:
    stopped HTTP pipelining with peer ffed16798470cab5 Jun 09 13:06:55
    controller-2 etcd[1876]: stopped streaming with peer ffed16798470cab5
    (stream MsgApp v2 reader) Jun 09 13:06:55 controller-2 etcd[1876]:
    stopped streaming with peer ffed16798470cab5 (stream Message reader)
    Jun 09 13:06:55 controller-2 etcd[1876]: stopped peer ffed16798470cab5
    Jun 09 13:06:55 controller-2 etcd[1876]: failed to find member
    f98dc20bce6225a0 in cluster 3e7cc799faffb625 Jun 09 13:06:55
    controller-2 etcd[1876]: forgot to set Type=notify in systemd service
    file?</p>
  </blockquote>
  
  <p>I indeed tried to start the etcd member using different commands but seems the etcd of controller-2 still stuck at unstarted state. May I know the reason of that? Any pointers would be highly appreciated! Thanks. </p>