<p>Our Kubernetes 1.6 cluster had certificates generated when the cluster was built on April 13th, 2017.</p>

<p>On December 13th, 2017, our cluster was upgraded to version 1.8, and new certificates were generated [apparently, an incomplete set of certificates].</p>

<p>On April 13th, 2018, we started seeing this message within our Kubernetes dashboard for api-server:</p>

<p><code>[authentication.go:64] Unable to authenticate the request due to an error: [x509: certificate has expired or is not yet valid, x509: certificate has expired or is not yet valid]</code></p>

<p>Tried pointing <strong>client-certificate</strong> &amp; <strong>client-key</strong> within <code>/etc/kubernetes/kubelet.conf</code> at the certificates generated on Dec 13th [<code>apiserver-kubelet-client.crt</code> and <code>apiserver-kubelet-client.crt</code>], but continue to see the above error.</p>

<p>Tried pointing <strong>client-certificate</strong> &amp; <strong>client-key</strong> within <code>/etc/kubernetes/kubelet.conf</code> at <strong>different</strong> certificates generated on Dec 13th [<code>apiserver.crt</code> and <code>apiserver.crt</code>] (I honestly don't understand the difference between these 2 sets of certs/keys), but continue to see the above error.</p>

<p>Tried pointing <strong>client-certificate</strong> &amp; <strong>client-key</strong> within <code>/etc/kubernetes/kubelet.conf</code> at non-existent files, and none of the kube* services would start, with <code>/var/log/syslog</code> complaining about this:</p>

<p><code>Apr 17 17:50:08 kuber01 kubelet[2422]: W0417 17:50:08.181326    2422 server.go:381] invalid kubeconfig: invalid configuration: [unable to read client-cert /tmp/this/cert/does/not/exist.crt for system:node:node01 due to open /tmp/this/cert/does/not/exist.crt: no such file or directory, unable to read client-key /tmp/this/key/does/not/exist.key for system:node:node01 due to open /tmp/this/key/does/not/exist.key: no such file or directory]</code></p>

<p>Any advice on how to overcome this error, or even troubleshoot it at a more granular level?  Was considering regenerating certificates for api-server (<code>kubeadm alpha phase certs apiserver</code>), based on instructions within <a href="https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-alpha/#cmd-phase-certs" rel="noreferrer">https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-alpha/#cmd-phase-certs</a> ... but not sure if I'd be doing more damage.</p>

<p>Relatively new to Kubernetes, and the gentleman who set this up is not available for consult ... any help is appreciated.  Thanks.</p>

<p>I think you need re-generate the apiserver certificate <code>/etc/kubernetes/pki/apiserver.crt</code> you can view current expire date like this.</p>
<pre><code>openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -text |grep ' Not '
            Not Before: Dec 20 14:32:00 2017 GMT
            Not After : Dec 20 14:32:00 2018 GMT
</code></pre>
<p>Here is the steps I used to regenerate the certificates on v1.11.5 cluster. compiled steps from here <a href="https://github.com/kubernetes/kubeadm/issues/581" rel="noreferrer">https://github.com/kubernetes/kubeadm/issues/581</a></p>
<hr />
<p><strong>to check all certificate expire date:</strong></p>
<pre><code>find /etc/kubernetes/pki/ -type f -name &quot;*.crt&quot; -print|egrep -v 'ca.crt$'|xargs -L 1 -t  -i bash -c 'openssl x509  -noout -text -in {}|grep After'
</code></pre>
<p><strong>Renew certificate on Master node.</strong></p>
<p>*) Renew certificate</p>
<pre><code>mv /etc/kubernetes/pki/apiserver.key /etc/kubernetes/pki/apiserver.key.old
mv /etc/kubernetes/pki/apiserver.crt /etc/kubernetes/pki/apiserver.crt.old
mv /etc/kubernetes/pki/apiserver-kubelet-client.crt /etc/kubernetes/pki/apiserver-kubelet-client.crt.old
mv /etc/kubernetes/pki/apiserver-kubelet-client.key /etc/kubernetes/pki/apiserver-kubelet-client.key.old
mv /etc/kubernetes/pki/front-proxy-client.crt /etc/kubernetes/pki/front-proxy-client.crt.old
mv /etc/kubernetes/pki/front-proxy-client.key /etc/kubernetes/pki/front-proxy-client.key.old


kubeadm alpha phase certs apiserver  --config /root/kubeadm-kubetest.yaml
kubeadm alpha phase certs apiserver-kubelet-client
kubeadm alpha phase certs front-proxy-client
 
mv /etc/kubernetes/pki/apiserver-etcd-client.crt /etc/kubernetes/pki/apiserver-etcd-client.crt.old
mv /etc/kubernetes/pki/apiserver-etcd-client.key /etc/kubernetes/pki/apiserver-etcd-client.key.old
kubeadm alpha phase certs  apiserver-etcd-client


mv /etc/kubernetes/pki/etcd/server.crt /etc/kubernetes/pki/etcd/server.crt.old
mv /etc/kubernetes/pki/etcd/server.key /etc/kubernetes/pki/etcd/server.key.old
kubeadm alpha phase certs  etcd-server --config /root/kubeadm-kubetest.yaml

mv /etc/kubernetes/pki/etcd/healthcheck-client.crt /etc/kubernetes/pki/etcd/healthcheck-client.crt.old
mv /etc/kubernetes/pki/etcd/healthcheck-client.key /etc/kubernetes/pki/etcd/healthcheck-client.key.old
kubeadm alpha phase certs  etcd-healthcheck-client --config /root/kubeadm-kubetest.yaml


mv /etc/kubernetes/pki/etcd/peer.crt /etc/kubernetes/pki/etcd/peer.crt.old
mv /etc/kubernetes/pki/etcd/peer.key /etc/kubernetes/pki/etcd/peer.key.old
kubeadm alpha phase certs  etcd-peer --config /root/kubeadm-kubetest.yaml

*)  Backup old configuration files
mv /etc/kubernetes/admin.conf /etc/kubernetes/admin.conf.old
mv /etc/kubernetes/kubelet.conf /etc/kubernetes/kubelet.conf.old
mv /etc/kubernetes/controller-manager.conf /etc/kubernetes/controller-manager.conf.old
mv /etc/kubernetes/scheduler.conf /etc/kubernetes/scheduler.conf.old

kubeadm alpha phase kubeconfig all  --config /root/kubeadm-kubetest.yaml

mv $HOME/.kube/config .$HOMEkube/config.old
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config
chmod 777 $HOME/.kube/config
export KUBECONFIG=.kube/config
</code></pre>
<p>Reboot the node and check the logs for etcd, kubeapi and kubelet.</p>
<p><strong>Note:</strong>
Remember to update your CI/CD job kubeconfig file. If you’re using helm command test that also.</p>
