<p>We did setup of k8s cluster on datacenter with 2 master and 5 worker node by using kubeadm for initialization  of cluster .Added  Cluster config for reference.</p>
<blockquote>
<p>kind: InitConfiguration
apiVersion: kubeadm.k8s.io/v1beta3
localAPIEndpoint:
advertiseAddress: VIP_ADDRESS
---
kind: ClusterConfiguration
apiVersion: kubeadm.k8s.io/v1beta3
kubernetesVersion: v1.22.0
controlPlaneEndpoint: &quot;MANAGEMENT_VIP_ADDRESS:6444&quot;
networking:
podSubnet: 192.168.0.0/16
---
kind: KubeletConfiguration
apiVersion: kubelet.config.k8s.io/v1beta1
address: 0.0.0.0
cgroupDriver: ADVERTISED_CGROUP_DRIVER_NAME
shutdownGracePeriod: 6m
shutdownGracePeriodCriticalPods: 4m</p>
</blockquote>
<p><strong>Version used for diffrent components.</strong>
etcd:3.5.0-0
kube-apiserver:v1.22
kube-controller-manager:v1.22
kubelet-1.22.1</p>
<p>Issue we do have is once we shutdown node  one, then complete server start misbehaving and nodes become read only on most of cases.Even we are not able to run kubectl command.</p>
<p>Getting Exception in kube api log:-</p>
<p>W0128 11:23:25.351294       1 clientconn.go:1326] [core] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 127.0.0.1  0 }. Err: connection error: desc = &quot;transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused&quot;. Reconnecting...
W0128 11:23:26.347009       1 clientconn.go:1326] [core] grpc: addrConn.createTransport failed to connect to {127.0.0.1:2379 127.0.0.1  0 }. Err: connection error: desc = &quot;transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused&quot;. Reconnecting...
W0128 11:23:26.352155       1 clientconn.go:1</p>

Mayank Singh
<p>Your etcd setup is not highly-available, it requires a minimum of 3 nodes to tolerate 1 node failure, otherwise etcd can corrupt data in a split-brain scenario</p>
<p>You should add at least one more node to etcd cluster (but 5 nodes is a recommended cluster size for production setups). Assuming that your master nodes are also running etcd, since you haven't specified otherwise, you should run 3 master nodes</p>
<p>See <a href="https://etcd.io/docs/v3.5/faq/#what-is-failure-tolerance" rel="nofollow noreferrer">https://etcd.io/docs/v3.5/faq/#what-is-failure-tolerance</a></p>
<blockquote>
<p>An etcd cluster operates so long as a member quorum can be established.</p>
</blockquote>
<blockquote>
<p>It is recommended to have an odd number of members in a cluster. An odd-size cluster tolerates the same number of failures as an even-size cluster but with fewer nodes. The difference can be seen by comparing even and odd sized clusters:</p>
</blockquote>
<div class="s-table-container">
<table class="s-table">
<thead>
<tr>
<th>Cluster Size</th>
<th>Majority</th>
<th>Failure Tolerance</th>
</tr>
</thead>
<tbody>
<tr>
<td>1</td>
<td>1</td>
<td>0</td>
</tr>
<tr>
<td>2</td>
<td>2</td>
<td>0</td>
</tr>
<tr>
<td>3</td>
<td>2</td>
<td>1</td>
</tr>
<tr>
<td>4</td>
<td>3</td>
<td>1</td>
</tr>
<tr>
<td>5</td>
<td>3</td>
<td>2</td>
</tr>
<tr>
<td>6</td>
<td>4</td>
<td>2</td>
</tr>
<tr>
<td>7</td>
<td>4</td>
<td>3</td>
</tr>
<tr>
<td>8</td>
<td>5</td>
<td>3</td>
</tr>
</tbody>
</table>
</div>