<p>I have the following scenario in the lab and would like to see if its possible to recover. The cluster is broken but very expected since I was testing how far I could go with breaking the cluster and still be able to recover.</p>
<p>Env:
Kubernetes 1.16.3
Kubespray</p>
<p>I was experimenting a bit and don't have any data on this cluster but I am still very curious if it's possible to recover. I have a healthy 3 node etcd cluster with the original configuration (all namespaces, workloads, configmaps etc). I don't have the original SSL certs for the control plane.</p>
<p>I removed all nodes from the cluster (kubeadm reset). I have original manifests and kubelet config and try to re-init master nodes. It is quite more successful than I thought it would be but not where I want it to be.
After successful kubeadm init, the kubelet and control plane containers start successfully but the corresponding pods are not created. I am able to use the kube API with kubectl and see the nodes, namespaces, deployments, etc.</p>
<p>In the kube-system namespace all daemonsets still exist but the pods won't start with the following message:</p>
<pre><code>49m         Warning   FailedCreate        daemonset/kube-proxy                                Error creating: Timeout: request did not complete within requested timeout
</code></pre>
<p>The kubelet logs the following re control plane pods</p>
<pre><code>Jul 21 22:30:02 k8s-master-4 kubelet[13791]: E0721 22:30:02.088787   13791 kubelet.go:1664] Failed creating a mirror pod for &quot;kube-scheduler-k8s-master-4_kube-system(3e128801ef687b022f6c8ae175c9c56d)&quot;: Timeout: request did not complete within requested timeout
Jul 21 22:30:53 k8s-master-4 kubelet[13791]: E0721 22:30:53.089517   13791 kubelet.go:1664] Failed creating a mirror pod for &quot;kube-controller-manager-k8s-master-4_kube-system(da5cfae13814fa171a320ce0605de98f)&quot;: Timeout: request did not complete within requested timeout
</code></pre>
<p>During kubeadm reset/init process I already have some steps so I can get to where I am now (delete serviceaccounts to reset the tokens, delete some configmaps (kuebadm etc))</p>
<p>My question is - is it possible to recover the control plane without the certs. And if its complicated but still possible process I would still like to know.</p>
<p>All help appreciated</p>
<p>Henro</p>

<blockquote>
<p>is it possible to recover the control plane without the certs.</p>
</blockquote>
<p>Yes, should be able to. The certs 🔏 are required but they don't have to be the very same ones that you created the cluster initially with. All the <a href="https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/" rel="nofollow noreferrer">certificates</a> including the <a href="https://kubernetes.io/docs/tasks/tls/manual-rotation-of-ca-certificates/" rel="nofollow noreferrer">CA</a> can be rotated across the board. The kubelet even supports <a href="https://kubernetes.io/docs/tasks/tls/certificate-rotation/" rel="nofollow noreferrer">certificate auto-rotation</a>. The configurations need to match everywhere though. Meaning the CA needs to be the same that created the CSRs and cert keys/certs need to be created from the same CSRs. 🔑</p>
<p>Also, all the components need to use the same CA and be able to authenticate with the API server (kube-controller-manager, kube-scheduler, etc) 🔐. I'm not entirely sure about the logs that you are seeing but it looks like the kube-controller-manager and kube-scheduler are not able to authenticate and join the cluster. So I would take a look at their cert configurations:</p>
<ul>
<li><code>/etc/kubernetes/kube-controller-manager.conf</code></li>
<li><code>/etc/kubernetes/kube-scheduler.conf</code></li>
</ul>
<p>Also, you would find every PKI component that you need to verify under <code>/etc/kubernetes/pki</code></p>
<p>✌️</p>
