<p>I made a Kubernetes cluster one year ago with an external etcd cluster (3 members).</p>

<p>A the time, I did not know that it was possible to make an etcd internal, so I made an external cluster and connected Kubernetes to it.</p>

<p>Now I am seeing that an internal cluster is a thing and it is a cleaner solution because the etcd nodes are updated when you update your Kubernetes cluster.</p>

<p>I can't find a clean solution to migrate an external etcd cluster to an internal cluster. I hope there is a solution with zero downtime. Do you know if it is possible please ?</p>

<p>Thank you for your response and have a nice day !</p>

Nurza
<p>The second solution I've mentioned in another answer is</p>
<h2>Growing and then shrinking etcd cluster</h2>
<p>The downside of this method is that etcd quorum size is temporary increased, and in case of several nodes failure, etcd cluster may <a href="https://medium.com/@zhimin.wen/testing-of-etcd-failure-936dc5833a62" rel="nofollow noreferrer">break</a>. To avoid it, you may want to remove one existing etcd cluster member before adding another one.</p>
<p>Here is the brief overview of the process:</p>
<ol>
<li>generate certificates for all additional members using etcd <code>ca.crt</code> and <code>ca.key</code> from existing etcd node folder (<code>/etc/kubernetes/pki/etcd/</code>).</li>
<li>add new member to the cluster using <code>etcdctl</code> command</li>
<li>create etcd config for new member</li>
<li>start new etcd member using new keys and config</li>
<li>check cluster health</li>
<li><strong>repeat steps 2-5</strong> until all required etcd nodes are added</li>
<li>remove one exessive etcd cluster member using etcdctl command</li>
<li>check cluster health</li>
<li><strong>repeat steps 7-8</strong> until the desired size of etcd cluster is achieved</li>
<li>Adjust all etcd.yaml files for all etcd cluster members.</li>
<li>Adjust etcd endpoints in all kube-apiserver.yaml manifests</li>
</ol>
<p>Another possible sequence:</p>
<ol>
<li>generate certificates for all additional members using etcd <code>ca.crt</code> and <code>ca.key</code> from existing etcd node folder (<code>/etc/kubernetes/pki/etcd/</code>).</li>
<li>remove one etcd cluster member using etcdctl command</li>
<li>add new member to the cluster using etcdctl command</li>
<li>create etcd config for new member</li>
<li>start new etcd member using new keys and config</li>
<li>check cluster health</li>
<li><strong>repeat steps 2-6</strong> until required etcd configuration is achieved</li>
<li>Adjust all <code>etcd.yaml</code> files for all etcd cluster members.</li>
<li>Adjust etcd endpoints in all kube-apiserver.yaml manifests</li>
</ol>
<hr />
<h3>How to generate certificates:</h3>
<ul>
<li>using kubeadm command (<a href="https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/setup-ha-etcd-with-kubeadm/#setting-up-the-cluster" rel="nofollow noreferrer">manual</a>)</li>
<li>using cfssl tool (<a href="https://github.com/kelseyhightower/kubernetes-the-hard-way/blob/master/docs/04-certificate-authority.md" rel="nofollow noreferrer">Kubernetes the hard way guide</a>)</li>
<li>using openssl (<a href="https://deliciousbrains.com/ssl-certificate-authority-for-local-https-development/" rel="nofollow noreferrer">link1</a>, <a href="https://jamielinux.com/docs/openssl-certificate-authority/" rel="nofollow noreferrer">link2</a>)</li>
</ul>
<p><strong>Note:</strong> If you have etcd cluster, you likely have etcd-CA certificate somewhere. Consider to use it along with the etcd-CA key to generate certificates for all additional etcd members.</p>
<p><strong>Note:</strong> In case you choose to generate certificates manually, usual Kubernetes certificates' parameters are:</p>
<ul>
<li>Signature Algorithm: sha256WithRSAEncryption</li>
<li>Public Key Algorithm: rsaEncryption</li>
<li>RSA Public-Key: (2048 bit)</li>
<li>CA certs age: 10 years</li>
<li>other certs age: 1 year</li>
</ul>
<p>You can check the content of the certificates using the following command:</p>
<pre><code>find /etc/kubernetes/pki/ -name *.crt | xargs -l  bash -c 'echo $0 ; openssl x509 -in $0 -text -noout'
</code></pre>
<hr />
<h3>How to <a href="https://docs.okd.io/latest/admin_guide/assembly_restore-etcd-quorum.html#removing-etcd-host_restore-etcd-quorum" rel="nofollow noreferrer">remove a member</a> from the etcd cluster</h3>
<p>(Please refer to my another answer, step 3, for variables and alias definitions)</p>
<pre><code>e3 member list

b67816d38b8e9d2, started, kube-ha-m3, https://10.128.0.12:2380, https://10.128.0.12:2379
3de72bd56f654b1c, started, kube-ha-m1, https://10.128.0.10:2380, https://10.128.0.10:2379
ac98ece88e3519b5, started, kube-etcd2, https://10.128.0.14:2380, https://10.128.0.14:2379
cfb0839e8cad4c8f, started, kube-ha-m2, https://10.128.0.11:2380, https://10.128.0.11:2379
eb9b83c725146b96, started, kube-etcd1, https://10.128.0.13:2380, https://10.128.0.13:2379
401a166c949e9584, started, kube-etcd3, https://10.128.0.15:2380, https://10.128.0.15:2379  # Let's remove this one

e2 member remove 401a166c949e9584
</code></pre>
<p>The member will shutdown instantly. To prevent further attempt of joining the cluster, move/delete etcd.yaml from /etc/kubernetes/manifests/ or shutdown etcd service on the etcd member node</p>
<hr />
<h3>How to <a href="https://docs.okd.io/latest/admin_guide/assembly_restore-etcd-quorum.html#adding-etcd-after-restoring_restore-etcd-quorum" rel="nofollow noreferrer">add a member</a> to the etcd cluster</h3>
<pre><code>e3 member add kube-etcd3 --peer-urls=&quot;https://10.128.0.16:2380&quot;
</code></pre>
<p>The output shows the parameters required to start the new etcd cluster member, e.g.:</p>
<pre><code>ETCD_NAME=&quot;kube-etcd3&quot;
ETCD_INITIAL_CLUSTER=&quot;kube-ha-m3=https://10.128.0.15:2380,kube-ha-m1=https://10.128.0.10:2380,kube-etcd2=https://10.128.0.14:2380,kube-ha-m2=https://10.128.0.11:2380,kube-etcd1=https://10.128.0.13:2380,kube-etcd3=https://10.128.0.16:2380&quot;
ETCD_INITIAL_ADVERTISE_PEER_URLS=&quot;https://10.128.0.16:2380&quot;
ETCD_INITIAL_CLUSTER_STATE=&quot;existing&quot;
</code></pre>
<p><strong>Note:</strong> <code>ETCD_INITIAL_CLUSTER</code> variable contains all existing etcd cluster members and also the new node. If you need to add several nodes it should be done one node at a time.</p>
<p><strong>Note:</strong> <code>All ETCD_INITIAL_*</code> variables and corresponded command line parameters only required for the first etcd Pod start. After successful addition of the node to the etcd cluster, these parameters are ignored and can be removed from startup configuration. All required information is stored in <code>/var/lib/etcd</code> folder in etcd database file.</p>
<p>The default <code>etcd.yaml</code> manifest could be generated using the following kubeadm commmand:</p>
<pre><code>kubeadm init phase etcd local
</code></pre>
<p>It's better to move <code>etcd.yaml</code> file from <code>/etc/kubernetes/manifests/</code> somewhere to make adjustments.</p>
<p>Also delete content of the <code>/var/lib/etcd</code> folder. It contains data of new etcd cluster, so it can't be used to add member to existing cluster.</p>
<p>Then it should be adjusted according to member add command output. (<code>--advertise-client-urls, -initial-advertise-peer-urls, --initial-cluster, --initial-cluster-state, --listen-client-urls, --listen-peer-urls</code>) E.g.:</p>
<pre><code>apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    component: etcd
    tier: control-plane
  name: etcd
  namespace: kube-system
spec:
  containers:
  - command:
    - etcd
    - --advertise-client-urls=https://10.128.0.16:2379
    - --cert-file=/etc/kubernetes/pki/etcd/server.crt
    - --client-cert-auth=true
    - --data-dir=/var/lib/etcd
    - --initial-advertise-peer-urls=https://10.128.0.16:2380
    - --initial-cluster=kube-ha-m3=https://10.128.0.15:2380,kube-ha-m1=https://10.128.0.10:2380,kube-etcd2=https://10.128.0.14:2380,kube-ha-m2=https://10.128.0.11:2380,kube-etcd1=https://10.128.0.13:2380,kube-etcd3=https://10.128.0.16:2380
    - --initial-cluster-state=existing
    - --key-file=/etc/kubernetes/pki/etcd/server.key
    - --listen-client-urls=https://10.128.0.16:2379
    - --listen-metrics-urls=http://127.0.0.1:2381
    - --listen-peer-urls=https://10.128.0.16:2380
    - --name=kube-etcd3
    - --peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt
    - --peer-client-cert-auth=true
    - --peer-key-file=/etc/kubernetes/pki/etcd/peer.key
    - --peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt
    - --snapshot-count=10000
    - --trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt
    image: k8s.gcr.io/etcd:3.3.10
    imagePullPolicy: IfNotPresent
    livenessProbe:
      failureThreshold: 8
      httpGet:
        host: 127.0.0.1
        path: /health
        port: 2381
        scheme: HTTP
      initialDelaySeconds: 15
      timeoutSeconds: 15
    name: etcd
    resources: {}
    volumeMounts:
    - mountPath: /var/lib/etcd
      name: etcd-data
    - mountPath: /etc/kubernetes/pki/etcd
      name: etcd-certs
  hostNetwork: true
  priorityClassName: system-cluster-critical
  volumes:
  - hostPath:
      path: /etc/kubernetes/pki/etcd
      type: DirectoryOrCreate
    name: etcd-certs
  - hostPath:
      path: /var/lib/etcd
      type: DirectoryOrCreate
    name: etcd-data
</code></pre>
<p>After saving the file, kubelet will restart etcd pod. Check etcd container logs to ensure it is joined to the cluster.</p>
<hr />
<h3>How to check cluster health</h3>
<pre><code>$ e2 cluster-health
member b67816d38b8e9d2 is healthy: got healthy result from https://10.128.0.15:2379
member 3de72bd56f654b1c is healthy: got healthy result from https://10.128.0.10:2379
member ac98ece88e3519b5 is healthy: got healthy result from https://10.128.0.14:2379
member cfb0839e8cad4c8f is healthy: got healthy result from https://10.128.0.11:2379
member eb9b83c725146b96 is healthy: got healthy result from https://10.128.0.13:2379
cluster is healthy

$ e2 member list
b67816d38b8e9d2: name=kube-ha-m3 peerURLs=https://10.128.0.15:2380 clientURLs=https://10.128.0.15:2379 isLeader=true
3de72bd56f654b1c: name=kube-ha-m1 peerURLs=https://10.128.0.10:2380 clientURLs=https://10.128.0.10:2379 isLeader=false
ac98ece88e3519b5: name=kube-etcd2 peerURLs=https://10.128.0.14:2380 clientURLs=https://10.128.0.14:2379 isLeader=false
cfb0839e8cad4c8f: name=kube-ha-m2 peerURLs=https://10.128.0.11:2380 clientURLs=https://10.128.0.11:2379 isLeader=false
eb9b83c725146b96: name=kube-etcd1 peerURLs=https://10.128.0.13:2380 clientURLs=https://10.128.0.13:2379 isLeader=false

$ e3 endpoint health
# the output includes only etcd members that are specified  in --endpoints cli option or corresponded environment variable. I've included only three out of five members
https://10.128.0.13:2379 is healthy: successfully committed proposal: took = 2.310436ms
https://10.128.0.15:2379 is healthy: successfully committed proposal: took = 1.795723ms
https://10.128.0.14:2379 is healthy: successfully committed proposal: took = 2.41462ms

$ e3 endpoint status
# the output includes only etcd members that are specified  in --endpoints cli option or corresponded environment variable. I've included only three out of five members
https://10.128.0.13:2379 is healthy: successfully committed proposal: took = 2.531676ms
https://10.128.0.15:2379 is healthy: successfully committed proposal: took = 1.285312ms
https://10.128.0.14:2379 is healthy: successfully committed proposal: took = 2.266932ms
</code></pre>
<hr />
<h3>How to check etcl Pod logs without using kubectl?</h3>
<p>If you run etcd member using kubelet only, you can check its log using the following command:</p>
<pre><code>docker logs `docker ps -a | grep etcd | grep -v pause | awk '{print $1}' | head -n1` 2&gt;&amp;1 | less
</code></pre>
<p><strong>Note:</strong> Usually, only one etcd Pod can be run on the same node at the same time, because it uses database in the host directory <code>/var/lib/etcd/</code> and it cannot be shared between two pods. Also etcd Pod uses node network interface to communicate with the etcd cluster.<br />
Of course, you can configure etcd Pod to use different host directory and use different host ports as a workaround, but the above command assumes that the only one etcd Pod is present on the node.</p>
