<p>since a couple of days and without any change in the environment one of the clusters running kubernetes 1.19.9 on-prem showed some errors regarding kubelet certificates.</p>
<p>A node is in NON-READY state due to an expired certificate. Investigating a bit i've found out that the CSR are in pending state. I can approve them manually but no issued at all.</p>
<p>I've tried to rejoin those nodes to the cluster but i face the same situation with the CSR approval.</p>
<p>Example:</p>
<pre><code>NAME        AGE     SIGNERNAME                                    REQUESTOR                        CONDITION
csr-4dc9x   3m28s   kubernetes.io/kube-apiserver-client-kubelet   system:node:vm-k8s-ctrl-prod-1   Pending
csr-4xljn   18m     kubernetes.io/kube-apiserver-client-kubelet   system:node:vm-k8s-wk-stage-9    Pending
csr-6jdmg   3m19s   kubernetes.io/kube-apiserver-client-kubelet   system:node:vm-k8s-wk-stage-6    Pending
csr-9lr8n   18m     kubernetes.io/kube-apiserver-client-kubelet   system:node:vm-k8s-wk-stage-6    Pending
csr-g2pjt   3m35s   kubernetes.io/kube-apiserver-client-kubelet   system:node:vm-k8s-ctrl-prod-2   Pending
</code></pre>
<p>CSR example:</p>
<pre><code>apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
  creationTimestamp: &quot;2021-08-08T10:10:19Z&quot;
  generateName: csr-
  managedFields:
  - apiVersion: certificates.k8s.io/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:generateName: {}
      f:spec:
        f:request: {}
        f:signerName: {}
        f:usages: {}
    manager: kubelet
    operation: Update
    time: &quot;2021-08-08T10:10:19Z&quot;
  name: csr-4dc9x
  resourceVersion: &quot;775314577&quot;
  selfLink: /apis/certificates.k8s.io/v1/certificatesigningrequests/csr-4dc9x
  uid: 8c51be15-4ec4-4dc7-8a7a-486e27c74607
spec:
  groups:
  - system:nodes
  - system:authenticated
  request: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlIN01JR2lBZ0VBTUVBeEZUQVRCZ05WQkFvVERITjVjM1JsYlRwdWIyUmxjekVuTUNVR0ExVUVBeE1lYzNsegpkR1Z0T201dlpHVTZkbTB0YXpoekxXTjBjbXd0Y0hKdlpDMHhNRmt3RXdZSEtvWkl6ajBDQVFZSUtvWkl6ajBECkFRY0RRZ0FFazNESFh2cTloVkZxZzB3bW5VeWd6Z3VGdmFRdDZFUkFCcHcrUmhRNHFCRlRqdkxTSGo3ZUxVK1oKT3JGaThaOGpYUjZqRE5nekVpUkxRQTloS1pxR0c2QUFNQW9HQ0NxR1NNNDlCQU1DQTBnQU1FVUNJUURObFJBcAphT0hFZWRteENDajZiK2tLMWJrNjVYVDc0aC9Nd1VCenVDSnBrUUlnU2F0U0Z3Rkp5ekNQaWtFZTRKQys0QStqClVtVUVWUzhlOWZRbkdXdjROTms9Ci0tLS0tRU5EIENFUlRJRklDQVRFIFJFUVVFU1QtLS0tLQo=
  signerName: kubernetes.io/kube-apiserver-client-kubelet
  usages:
  - digital signature
  - key encipherment
  - client auth
  username: system:node:vm-k8s-ctrl-prod-1
status: {}
</code></pre>
<p>Did anyone face the same situation? i've checked all the certificates in the cluster and everything looks good to me.</p>
<pre><code>
CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Jun 10, 2022 22:17 UTC   306d                                    no
apiserver                  Jun 10, 2022 22:16 UTC   306d            ca                      no
apiserver-kubelet-client   Jun 10, 2022 22:16 UTC   306d            ca                      no
controller-manager.conf    Jun 10, 2022 22:17 UTC   306d                                    no
front-proxy-client         Jun 10, 2022 22:16 UTC   306d            front-proxy-ca          no
scheduler.conf             Jun 10, 2022 22:17 UTC   306d                                    no

CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Apr 07, 2029 17:39 UTC   7y              no
front-proxy-ca          Apr 07, 2029 17:39 UTC   7y              no
</code></pre>
<p>Thanks in advance</p>

trookam
<p>just in case anyone else face this situation. The issue was a legacy configuration for kubelet on the master nodes.</p>
<p><a href="https://serverfault.com/questions/1065444/how-can-i-find-which-kubernetes-certificate-has-expired">https://serverfault.com/questions/1065444/how-can-i-find-which-kubernetes-certificate-has-expired</a></p>
<p>reconfiguring manually the kubelet.conf on the controllers and restarting the control-plane, fixed the issue.</p>
<p>Thanks</p>
