<p>My cluster certificates are expired and now I cannot execute any kubectls commands.</p>
<pre><code>root@node1:~# kubectl get ns
Unable to connect to the server: x509: certificate has expired or is not yet valid
root@node1:~# 
</code></pre>
<p>I have created this cluster using Kubespray , kubeadm version is v1.16.3 and kubernetesVersion v1.16.3</p>
<pre><code>root@node1:~# kubeadm alpha certs check-expiration
failed to load existing certificate apiserver-etcd-client: open /etc/kubernetes/pki/apiserver-etcd-client.crt: no such file or directory
To see the stack trace of this error execute with --v=5 or higher
root@node1:~# 
</code></pre>
<p>And it is found that apiserver-etcd-client.crt and apiserver-etcd-client.key files are missing on /etc/kubernetes/pki directory.</p>
<pre><code>root@node1:/etc/kubernetes/pki# ls -ltr
total 72
-rw------- 1 root root 1679 Jan 24 2020 ca.key
-rw-r--r-- 1 root root 1025 Jan 24 2020 ca.crt
-rw-r----- 1 root root 1679 Jan 24 2020 apiserver.key.old
-rw-r----- 1 root root 1513 Jan 24 2020 apiserver.crt.old
-rw------- 1 root root 1679 Jan 24 2020 apiserver.key
-rw-r--r-- 1 root root 1513 Jan 24 2020 apiserver.crt
-rw------- 1 root root 1675 Jan 24 2020 apiserver-kubelet-client.key
-rw-r--r-- 1 root root 1099 Jan 24 2020 apiserver-kubelet-client.crt
-rw-r----- 1 root root 1675 Jan 24 2020 apiserver-kubelet-client.key.old
-rw-r----- 1 root root 1099 Jan 24 2020 apiserver-kubelet-client.crt.old
-rw------- 1 root root 1679 Jan 24 2020 front-proxy-ca.key
-rw-r--r-- 1 root root 1038 Jan 24 2020 front-proxy-ca.crt
-rw-r----- 1 root root 1675 Jan 24 2020 front-proxy-client.key.old
-rw-r----- 1 root root 1058 Jan 24 2020 front-proxy-client.crt.old
-rw------- 1 root root 1675 Jan 24 2020 front-proxy-client.key
-rw-r--r-- 1 root root 1058 Jan 24 2020 front-proxy-client.crt
-rw------- 1 root root 451 Jan 24 2020 sa.pub
-rw------- 1 root root 1679 Jan 24 2020 sa.key
root@node1:/etc/kubernetes/pki#
</code></pre>
<p>I have tried the following command but nothing is worked and showing errors:</p>
<pre><code>#sudo kubeadm alpha certs renew all
#kubeadm alpha phase certs apiserver-etcd-client
#kubeadm alpha certs apiserver-etcd-client --config /etc/kubernetes/kubeadm-config.yaml
</code></pre>
<p>Kubespray command:</p>
<pre><code>#ansible-playbook -i inventory/mycluster/hosts.yaml --become --become-user=root cluster.yml
</code></pre>
<p>The above command ended with the below error:</p>
<p>FAILED! =&gt; {&quot;attempts&quot;: 5, &quot;changed&quot;: true, &quot;cmd&quot;: [&quot;/usr/local/bin/kubeadm&quot;, &quot;--kubeconfig&quot;, &quot;/etc/kubernetes/admin.conf&quot;, &quot;token&quot;, &quot;create&quot;], &quot;delta&quot;: &quot;0:01:15.058756&quot;, &quot;end&quot;: &quot;2021-02-05 13:32:51.656901&quot;, &quot;msg&quot;: &quot;non-zero return code&quot;, &quot;rc&quot;: 1, &quot;start&quot;: &quot;2021-02-05 13:31:36.598145&quot;, &quot;stderr&quot;: &quot;timed out waiting for the condition\nTo see the stack trace of this error execute with --v=5 or higher&quot;, &quot;stderr_lines&quot;: [&quot;timed out waiting for the condition&quot;, &quot;To see the stack trace of this error execute with --v=5 or higher&quot;], &quot;stdout&quot;: &quot;&quot;, &quot;stdout_lines&quot;: []}</p>
<pre><code># cat /etc/kubernetes/kubeadm-config.yaml 
apiVersion: kubeadm.k8s.io/v1beta2
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: master1_IP
  bindPort: 6443
certificateKey: xxx
nodeRegistration:
  name: node1
  taints:
  - effect: NoSchedule
    key: node-role.kubernetes.io/master
  criSocket: /var/run/dockershim.sock
---
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
clusterName: cluster.local
etcd:
  external:
      endpoints:
      - https://master1:2379
      - https://master2:2379
      - https://master3:2379
      caFile: /etc/ssl/etcd/ssl/ca.pem
      certFile: /etc/ssl/etcd/ssl/node-node1.pem
      keyFile: /etc/ssl/etcd/ssl/node-node1-key.pem
dns:
  type: CoreDNS
  imageRepository: docker.io/coredns
  imageTag: 1.6.0
networking:
  dnsDomain: cluster.local
  serviceSubnet: IP/18
  podSubnet: IP/18
kubernetesVersion: v1.16.3
controlPlaneEndpoint: master1_IP:6443
certificatesDir: /etc/kubernetes/ssl
imageRepository: gcr.io/google-containers
apiServer:
</code></pre>

<p>First you need to renew expired certificates, use <code>kubeadm</code> to do this:</p>
<pre><code>kubeadm alpha certs renew apiserver
kubeadm alpha certs renew apiserver-kubelet-client
kubeadm alpha certs renew front-proxy-client
</code></pre>
<p>Next generate new <code>kubeconfig</code> files:</p>
<pre><code>kubeadm alpha kubeconfig user --client-name kubernetes-admin --org system:masters &gt; /etc/kubernetes/admin.conf
kubeadm alpha kubeconfig user --client-name system:kube-controller-manager &gt; /etc/kubernetes/controller-manager.conf
# instead of $(hostname) you may need to pass the name of the master node as in &quot;/etc/kubernetes/kubelet.conf&quot; file.
kubeadm alpha kubeconfig user --client-name system:node:$(hostname) --org system:nodes &gt; /etc/kubernetes/kubelet.conf 
kubeadm alpha kubeconfig user --client-name system:kube-scheduler &gt; /etc/kubernetes/scheduler.conf
</code></pre>
<p>Copy new <code>kubernetes-admin</code> <code>kubeconfig</code> file:</p>
<pre><code>cp /etc/kubernetes/admin.conf ~/.kube/config
</code></pre>
<p>Finally you need to restart: <code>kube-apiserver</code>, <code>kube-controller-manager</code> and <code>kube-scheduler</code>. You can use below commands or just restart master node:</p>
<pre><code>sudo kill -s SIGHUP $(pidof kube-apiserver)
sudo kill -s SIGHUP $(pidof kube-controller-manager)
sudo kill -s SIGHUP $(pidof kube-scheduler)
</code></pre>
<p>Additionally you can find more information on <a href="https://github.com/kubernetes/kubeadm/issues/581" rel="nofollow noreferrer">github</a> and <a href="https://github.com/kubernetes/kubeadm/issues/581#issuecomment-471575078" rel="nofollow noreferrer">this answer</a> may be of great help to you.</p>
