<p>I have a standalone Kuberenets cluster installed on some physical RHEL machine.</p>
<p>I'm experiencing recurring crashes of <code>etcd</code> and <code>kube-apiserver</code> containers. From their logs, I managed to guess that I need to tune etcd to better perform in this environment.</p>
<p>The following guide references how to tune etcd: <a href="https://etcd.io/docs/v3.4.0/tuning/" rel="nofollow noreferrer">https://etcd.io/docs/v3.4.0/tuning/</a></p>
<p>However, I'm not sure how it can be done in an existing cluster. Is it a kubernetes native component? Should I patch its deployment? Its quite hard with etcd itself being down.</p>
<p>Error logs snippets:</p>
<p><strong>etcd</strong></p>
<pre><code>[WARNING] Deprecated '--logger=capnslog' flag is set; use '--logger=zap' flag instead
2021-03-09 08:09:12.996848 I | etcdmain: etcd Version: 3.4.13
2021-03-09 08:09:12.996909 I | etcdmain: Git SHA: ae9734ed2
2021-03-09 08:09:12.996915 I | etcdmain: Go Version: go1.12.17
2021-03-09 08:09:12.996926 I | etcdmain: Go OS/Arch: linux/amd64
2021-03-09 08:09:12.996932 I | etcdmain: setting maximum number of CPUs to 24, total number of available CPUs is 24
2021-03-09 08:09:12.996939 N | etcdmain: failed to detect default host (could not find default route)
2021-03-09 08:09:12.997017 N | etcdmain: the server is already initialized as member before, starting as etcd member...
[WARNING] Deprecated '--logger=capnslog' flag is set; use '--logger=zap' flag instead
2021-03-09 08:09:12.997071 I | embed: peerTLS: cert = /etc/kubernetes/pki/etcd/peer.crt, key = /etc/kubernetes/pki/etcd/peer.key, trusted-ca = /etc/kubernetes/pki/etcd/ca.crt, client-cert-auth = true, crl-file = 
2021-03-09 08:09:13.004253 I | embed: name = hostname-acme.com
2021-03-09 08:09:13.004268 I | embed: data dir = /var/lib/etcd
2021-03-09 08:09:13.004276 I | embed: member dir = /var/lib/etcd/member
2021-03-09 08:09:13.004284 I | embed: heartbeat = 100ms
2021-03-09 08:09:13.004292 I | embed: election = 1000ms
2021-03-09 08:09:13.004297 I | embed: snapshot count = 10000
2021-03-09 08:09:13.004339 I | embed: advertise client URLs = https://10.43.16.56:2379
2021-03-09 08:09:13.004351 I | embed: initial advertise peer URLs = https://10.43.16.56:2380
2021-03-09 08:09:13.004358 I | embed: initial cluster = 
2021-03-09 08:09:13.004413 W | pkg/fileutil: check file permission: directory &quot;/var/lib/etcd&quot; exist, but the permission is &quot;drwxrwxrwx&quot;. The recommended permission is &quot;-rwx------&quot; to prevent possible unprivileged access to the data.
2021-03-09 08:09:14.066995 I | etcdserver: recovered store from snapshot at index 1140183
2021-03-09 08:09:14.259395 I | mvcc: restore compact to 935600
2021-03-09 08:09:14.538404 I | etcdserver: restarting member fe007e8a424c7486 in cluster 59c13b0ba56e3f74 at commit index 1141310
raft2021/03/09 08:09:14 INFO: fe007e8a424c7486 switched to configuration voters=(18302768017916589190)
raft2021/03/09 08:09:14 INFO: fe007e8a424c7486 became follower at term 1050
raft2021/03/09 08:09:14 INFO: newRaft fe007e8a424c7486 [peers: [fe007e8a424c7486], term: 1050, commit: 1141310, applied: 1140183, lastindex: 1141310, lastterm: 1050]
2021-03-09 08:09:14.539625 I | etcdserver/api: enabled capabilities for version 3.4
2021-03-09 08:09:14.539659 I | etcdserver/membership: added member fe007e8a424c7486 [https://10.43.16.56:2380] to cluster 59c13b0ba56e3f74 from store
2021-03-09 08:09:14.539671 I | etcdserver/membership: set the cluster version to 3.4 from store
2021-03-09 08:09:14.650895 W | auth: simple token is not cryptographically signed
2021-03-09 08:09:14.784236 I | mvcc: restore compact to 935600
2021-03-09 08:09:14.918183 I | etcdserver: starting server... [version: 3.4.13, cluster version: 3.4]
2021-03-09 08:09:14.918532 I | etcdserver: fe007e8a424c7486 as single-node; fast-forwarding 9 ticks (election ticks 10)
2021-03-09 08:09:14.922068 I | embed: ClientTLS: cert = /etc/kubernetes/pki/etcd/server.crt, key = /etc/kubernetes/pki/etcd/server.key, trusted-ca = /etc/kubernetes/pki/etcd/ca.crt, client-cert-auth = true, crl-file = 
2021-03-09 08:09:14.922089 I | embed: listening for peers on 10.43.16.56:2380
2021-03-09 08:09:14.922328 I | embed: listening for metrics on http://127.0.0.1:2381
2021-03-09 08:09:14.925808 W | etcdserver: failed to apply request &quot;header:&lt;ID:8396530591132430964 &gt; lease_revoke:&lt;id:74867815fe26322c&gt;&quot; with response &quot;size:30&quot; took (23.834µs) to execute, err is lease not found
2021-03-09 08:09:14.925900 W | etcdserver: failed to apply request &quot;header:&lt;ID:8396530591132430966 &gt; lease_revoke:&lt;id:74867815fe263251&gt;&quot; with response &quot;size:30&quot; took (18.438µs) to execute, err is lease not found
2021-03-09 08:09:14.925961 W | etcdserver: failed to apply request &quot;header:&lt;ID:8396530591132430968 &gt; lease_revoke:&lt;id:74867815fe26322c&gt;&quot; with response &quot;size:30&quot; took (26.391µs) to execute, err is lease not found
2021-03-09 08:09:14.926062 W | etcdserver: failed to apply request &quot;header:&lt;ID:8396530591132430971 &gt; lease_revoke:&lt;id:74867815fe263251&gt;&quot; with response &quot;size:30&quot; took (18.063µs) to execute, err is lease not found
2021-03-09 08:09:14.926114 W | etcdserver: failed to apply request &quot;header:&lt;ID:8396530591132430973 &gt; lease_revoke:&lt;id:74867815fe26322c&gt;&quot; with response &quot;size:30&quot; took (26.452µs) to execute, err is lease not found
2021-03-09 08:09:14.926194 W | etcdserver: failed to apply request &quot;header:&lt;ID:8396530591132430975 &gt; lease_revoke:&lt;id:74867815fe263251&gt;&quot; with response &quot;size:30&quot; took (17.427µs) to execute, err is lease not found
2021-03-09 08:09:14.926282 W | etcdserver: failed to apply request &quot;header:&lt;ID:8396530591132430979 &gt; lease_revoke:&lt;id:74867815fe26322c&gt;&quot; with response &quot;size:30&quot; took (15.211µs) to execute, err is lease not found
2021-03-09 08:09:14.926333 W | etcdserver: failed to apply request &quot;header:&lt;ID:8396530591132430980 &gt; lease_revoke:&lt;id:74867815fe263251&gt;&quot; with response &quot;size:30&quot; took (24.87µs) to execute, err is lease not found
raft2021/03/09 08:09:15 INFO: fe007e8a424c7486 is starting a new election at term 1050
raft2021/03/09 08:09:15 INFO: fe007e8a424c7486 became candidate at term 1051
raft2021/03/09 08:09:15 INFO: fe007e8a424c7486 received MsgVoteResp from fe007e8a424c7486 at term 1051
raft2021/03/09 08:09:15 INFO: fe007e8a424c7486 became leader at term 1051
raft2021/03/09 08:09:15 INFO: raft.node: fe007e8a424c7486 elected leader fe007e8a424c7486 at term 1051
2021-03-09 08:09:15.804920 I | etcdserver: published {Name:hostname-acme.com ClientURLs:[https://10.43.16.56:2379]} to cluster 59c13b0ba56e3f74
2021-03-09 08:09:15.804960 I | embed: ready to serve client requests
2021-03-09 08:09:15.805085 I | embed: ready to serve client requests
2021-03-09 08:09:15.806545 I | embed: serving client requests on 10.43.16.56:2379
2021-03-09 08:09:15.806594 I | embed: serving client requests on 127.0.0.1:2379
...
2021-03-09 08:15:50.701225 W | etcdserver: read-only range request &quot;key:\&quot;/registry/events/default/xxx-pvc.166a9e8623334d39\&quot; &quot; with result &quot;error:context canceled&quot; took too long (5.378372536s) to execute
WARNING: 2021/03/09 08:15:50 grpc: Server.processUnaryRPC failed to write status: connection error: desc = &quot;transport is closing&quot;
2021-03-09 08:15:52.165933 W | etcdserver: read-only range request &quot;key:\&quot;/registry/health\&quot; &quot; with result &quot;error:context deadline exceeded&quot; took too long (2.000041013s) to execute
WARNING: 2021/03/09 08:15:52 grpc: Server.processUnaryRPC failed to write status: connection error: desc = &quot;transport is closing&quot;
2021-03-09 08:15:52.581845 W | etcdserver: read-only range request &quot;key:\&quot;/registry/health\&quot; &quot; with result &quot;error:context deadline exceeded&quot; took too long (2.000024386s) to execute
WARNING: 2021/03/09 08:15:52 grpc: Server.processUnaryRPC failed to write status: connection error: desc = &quot;transport is closing&quot;
2021-03-09 08:15:52.879773 W | etcdserver: failed to revoke 74867816086e2e90 (&quot;etcdserver: request timed out&quot;)
</code></pre>
<p><strong>kube-apiserver</strong></p>
<pre><code>I0309 08:27:06.013263       1 server.go:163] Version: v1.19.7
...
...
E0309 08:31:14.921841       1 storage_rbac.go:317] unable to reconcile rolebinding.rbac.authorization.k8s.io/system:controller:token-cleaner in kube-system: Get &quot;https://[::1]:6443/apis/rbac.authorization.k8s.io/v1/namespaces/kube-system/rolebindings/system:controller:token-cleaner&quot;: dial tcp [::1]:6443: connect: connection refused
E0309 08:31:14.922948       1 storage_rbac.go:317] unable to reconcile rolebinding.rbac.authorization.k8s.io/system:controller:bootstrap-signer in kube-public: Get &quot;https://[::1]:6443/apis/rbac.authorization.k8s.io/v1/namespaces/kube-public/rolebindings/system:controller:bootstrap-signer&quot;: dial tcp [::1]:6443: connect: connection refused
...
W0309 08:31:33.190649       1 clientconn.go:1223] grpc: addrConn.createTransport failed to connect to {https://127.0.0.1:2379  &lt;nil&gt; 0 &lt;nil&gt;}. Err :connection error: desc = &quot;transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused&quot;. Reconnecting...
W0309 08:31:33.255700       1 clientconn.go:1223] grpc: addrConn.createTransport failed to connect to {https://127.0.0.1:2379  &lt;nil&gt; 0 &lt;nil&gt;}. Err :connection error: desc = &quot;transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused&quot;. Reconnecting...
W0309 08:31:33.565370       1 controller.go:193] RemoveEndpoints() timed out
</code></pre>

Mugen
<p>Indeed, you will get the YAML of the <code>Deployment</code>, and make the changes.</p>
<p>If you installed the cluster with <code>kubeadm</code>, the file will be under <code>/etc/kubernetes/manifests/</code>, and once you make the changes, it will automatically be re-deployed.</p>
