<p>In my project we have etcd DB deployed on Kubernetes (this etcd is for application use, separate from the Kubernetes etcd) on on-prem. So I deployed it using the bitnami helm chart as a statefulset. Initially, at the time of deployment, the number of replicas was 1 as we wanted a single instance of etcd DB earlier.</p>
<p>The real problem started when we scaled it up to 3. I updated configuration to scale it up by updating the ETCD_INITIAL_CLUSTER with two new members DNS name:</p>
<pre><code>etcd-0=http://etcd-0.etcd-headless.wallet.svc.cluster.local:2380,etcd-1=http://etcd-1.etcd-headless.wallet.svc.cluster.local:2380,etcd-2=http://etcd-2.etcd-headless.wallet.svc.cluster.local:2380
</code></pre>
<p>Now when I go inside any of etcd pod and run <em>etcdctl member list</em> I only get a list of member and none of them is selected as leader, which is wrong. One among three should be the leader.</p>
<p>Also after running for some time these pods start giving heartbeat exceeds error and server overload error:</p>
<pre class="lang-sh prettyprint-override"><code>W |  etcdserver: failed to send out heartbeat on time (exceeded the 950ms timeout for 593.648512ms, to a9b7b8c4e027337a
W | etcdserver: server is likely overloaded
W | wal: sync duration of 2.575790761s, expected less than 1s
</code></pre>
<p>I changed the heartbeat default value accordingly, the number of errors decreased but still, I get a few heartbeat exceed errors along with others.</p>
<p>Not sure what is the problem here, is it the i/o that's causing the problem? If yes I am not sure how to be sure.</p>
<p>Will really appreciate any help on this.</p>

<p>I don't think 🤔 the heartbeats are the main problem, it also seems 👀 the logs that you are seeing are Warning logs. So it's possible that some heartbeats are missed here and there but your nodes are node(s) are not crashing or mirroring.</p>
<p>It's likely that you changed the replica numbers and your new replicas are not joining the cluster. So, I would recommend following this <a href="https://github.com/etcd-io/etcd/blob/master/Documentation/op-guide/runtime-configuration.md#remove-a-member" rel="nofollow noreferrer">guide</a> for you to add the new members to the cluster. Basically with <code>etcdctl</code> something like this:</p>
<pre class="lang-sh prettyprint-override"><code>etcdctl member add node2 --peer-urls=http://node1:2380
etcdctl member add node3 --peer-urls=http://node1:2380,http://node2:2380
</code></pre>
<p>Note that you will have to run these commands in a pod that has access to all your etcd nodes in your cluster.</p>
<p>You could also consider managing your etcd cluster with the <a href="https://operatorhub.io/operator/etcd" rel="nofollow noreferrer">etcd operator</a> 🔧 which should be able to take care of the scaling and removal/addition of nodes.</p>
<p>✌️</p>
