<p><code>rke --debug up --config cluster.yml</code></p>
<p>fails with health checks on etcd hosts with error:</p>
<blockquote>
<p>DEBU[0281] [etcd] failed to check health for etcd host [x.x.x.x]: failed to get /health for host [x.x.x.x]: Get &quot;https://x.x.x.x:2379/health&quot;: remote error: tls: bad certificate</p>
</blockquote>
<p>Checking etcd healthchecks</p>
<pre><code>for endpoint in $(docker exec etcd /bin/sh -c &quot;etcdctl member list | cut -d, -f5&quot;); do
  echo &quot;Validating connection to ${endpoint}/health&quot;;
  curl -w &quot;\n&quot; --cacert $(docker exec etcd printenv ETCDCTL_CACERT) --cert $(docker exec etcd printenv ETCDCTL_CERT) --key $(docker exec etcd printenv ETCDCTL_KEY) &quot;${endpoint}/health&quot;;
done

Running on that master node
Validating connection to https://x.x.x.x:2379/health
{&quot;health&quot;:&quot;true&quot;}
Validating connection to https://x.x.x.x:2379/health
{&quot;health&quot;:&quot;true&quot;}
Validating connection to https://x.x.x.x:2379/health
{&quot;health&quot;:&quot;true&quot;}
Validating connection to https://x.x.x.x:2379/health
{&quot;health&quot;:&quot;true&quot;}
</code></pre>
<pre><code>you can run it manually and see if it responds correctly
curl -w &quot;\n&quot; --cacert /etc/kubernetes/ssl/kube-ca.pem --cert /etc/kubernetes/ssl/kube-etcd-x-x-x-x.pem --key /etc/kubernetes/ssl/kube-etcd-x-x-x-x-key.pem https://x.x.x.x:2379/health
</code></pre>
<p>Checking my self signed certificates hashes</p>
<pre><code># md5sum /etc/kubernetes/ssl/kube-ca.pem
f5b358e771f8ae8495c703d09578eb3b  /etc/kubernetes/ssl/kube-ca.pem

# for key in $(cat /home/kube/cluster.rkestate | jq -r '.desiredState.certificatesBundle | keys[]'); do echo $(cat /home/kube/cluster.rkestate | jq -r --arg key $key '.desiredState.certificatesBundle[$key].certificatePEM' | sed '$ d' | md5sum) $key; done | grep kube-ca
f5b358e771f8ae8495c703d09578eb3b - kube-ca
</code></pre>
<pre><code>versions on my master node
Debian GNU/Linux 10
rke version v1.3.1
docker version Version: 20.10.8
kubectl v1.21.5
v1.21.5-rancher1-1
</code></pre>
<p>I think my <code>cluster.rkestate</code> gone bad, are there any other locations where rke tool checks for certificates?
Currently I cannot do anything with this production cluster, and want to avoid downtime. I experimented on testing cluster different scenarios, I could do as last resort to recreate the cluster from scratch, but maybe I can still fix it...
<code>rke remove</code> &amp;&amp; <code>rke up</code></p>

arnittocrab
<p><code>rke util get-state-file</code> helped me to reconstruct bad cluster.rkestate file
and I was able to successfully <code>rke up</code> and add new master node to fix whole situation.</p>
