<p>Currently I am using a script to renew Kubernetes certificates before they expire. But this is a manual process. I have to monitor expiration dates carefully and run this script beforehand. What's the recommended way to update all control plane certificates automatically without updating control plane? Do kubelet's --rotate* flags rotate all components (e.g. controller) or it is just for kubelet? PS: Kubernetes cluster was created with kubeadm.</p>

Baris Simsek
<p>Answering following question:</p>
<blockquote>
<p>What's the recommended way to update all control plane certificates automatically without updating control plane</p>
</blockquote>
<p>According to the k8s docs and best practices the best practice is to use &quot;Automatic certificate renewal&quot; with control plane upgrade:</p>
<blockquote>
<h3>Automatic certificate renewal</h3>
<p>This feature is designed for addressing the simplest use cases; if you don't have specific requirements on certificate renewal and perform Kubernetes version upgrades regularly (less than 1 year in between each upgrade), kubeadm will take care of keeping your cluster up to date and reasonably secure.</p>
<p><strong>Note:</strong> It is a best practice to upgrade your cluster frequently in order to stay secure.</p>
<p>-- <em><a href="https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/#automatic-certificate-renewal" rel="noreferrer">Kubernetes.io: Administer cluster: Kubeadm certs: Automatic certificate renewal</a></em></p>
</blockquote>
<p>Why this is the recommended way:</p>
<p>From the best practices standpoint you should be upgrading your <code>control-plane</code> to patch vulnerabilities, add features and use the version that is currently supported.</p>
<p>Each <code>control-plane</code> upgrade will renew the certificates as described (defaults to <code>true</code>):</p>
<ul>
<li><code>$ kubeadm upgrade apply --help</code></li>
</ul>
<pre class="lang-sh prettyprint-override"><code>--certificate-renewal    Perform the renewal of certificates used by component changed during upgrades. (default true)
</code></pre>
<p>You can also check the expiration of the <code>control-plane</code> certificates by running:</p>
<ul>
<li><code>$ kubeadm certs check-expiration</code></li>
</ul>
<pre class="lang-sh prettyprint-override"><code>[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
 
CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 May 30, 2022 13:36 UTC   364d                                    no      
apiserver                  May 30, 2022 13:36 UTC   364d            ca                      no      
apiserver-etcd-client      May 30, 2022 13:36 UTC   364d            etcd-ca                 no      
apiserver-kubelet-client   May 30, 2022 13:36 UTC   364d            ca                      no      
controller-manager.conf    May 30, 2022 13:36 UTC   364d                                    no      
etcd-healthcheck-client    May 30, 2022 13:36 UTC   364d            etcd-ca                 no      
etcd-peer                  May 30, 2022 13:36 UTC   364d            etcd-ca                 no      
etcd-server                May 30, 2022 13:36 UTC   364d            etcd-ca                 no      
front-proxy-client         May 30, 2022 13:36 UTC   364d            front-proxy-ca          no      
scheduler.conf             May 30, 2022 13:36 UTC   364d                                    no      
 
CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      May 28, 2031 13:36 UTC   9y              no      
etcd-ca                 May 28, 2031 13:36 UTC   9y              no      
front-proxy-ca          May 28, 2031 13:36 UTC   9y              no  
</code></pre>
<blockquote>
<p><strong>A side note!</strong></p>
<p><code>kubelet.conf</code> is not included in the list above because <code>kubeadm</code> configures <code>kubelet</code> for automatic certificate renewal.</p>
</blockquote>
<p>From what it can be seen by default:</p>
<blockquote>
<ul>
<li>Client certificates generated by <a href="https://kubernetes.io/docs/reference/setup-tools/kubeadm/" rel="noreferrer">kubeadm</a> expire after 1 year.</li>
<li>CA created by <a href="https://kubernetes.io/docs/reference/setup-tools/kubeadm/" rel="noreferrer">kubeadm</a> are set to expire after 10 years.</li>
</ul>
</blockquote>
<hr />
<p>There are other features that allows you to rotate the certificates in a &quot;semi automatic&quot; way.</p>
<p>You can opt for a manual certificate renewal with the:</p>
<ul>
<li><code>$ kubeadm certs renew</code></li>
</ul>
<p>where you can automatically (with the command) renew the specified (or all) certificates:</p>
<ul>
<li><code>$ kubeadm certs renew all</code></li>
</ul>
<pre><code>[renew] Reading configuration from the cluster...
[renew] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
 
certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed
certificate for serving the Kubernetes API renewed
certificate the apiserver uses to access etcd renewed
certificate for the API server to connect to kubelet renewed
certificate embedded in the kubeconfig file for the controller manager to use renewed
certificate for liveness probes to healthcheck etcd renewed
certificate for etcd nodes to communicate with each other renewed
certificate for serving etcd renewed
certificate for the front proxy client renewed
certificate embedded in the kubeconfig file for the scheduler manager to use renewed
 
Done renewing certificates. You must restart the kube-apiserver, kube-controller-manager, kube-scheduler and etcd, so that they can use the new certificates.
</code></pre>
<p>Please take a specific look on the output:</p>
<pre><code>You must restart the kube-apiserver, kube-controller-manager, kube-scheduler and etcd, so that they can use the new certificates.
</code></pre>
<p>As pointed, you will need to restart the components of your <code>control-plane</code> to use new certificate but remember:</p>
<ul>
<li><code>$ kubectl delete pod -n kube-system kube-scheduler-ubuntu</code> <strong>will not work</strong>.</li>
</ul>
<p>You will need to restart the docker container responsible for the component:</p>
<ul>
<li><code>$ docker ps | grep -i &quot;scheduler&quot;</code></li>
<li><code>$ docker restart 8c361562701b</code> (example)</li>
</ul>
<pre><code>8c361562701b   38f903b54010             &quot;kube-scheduler --au…&quot;   11 minutes ago      Up 11 minutes                k8s_kube-scheduler_kube-scheduler-ubuntu_kube-system_dbb97c1c9c802fa7cf2ad7d07938bae9_5
b709e8fb5e6c   k8s.gcr.io/pause:3.4.1   &quot;/pause&quot;                 About an hour ago   Up About an hour             k8s_POD_kube-scheduler-ubuntu_kube-system_dbb97c1c9c802fa7cf2ad7d07938bae9_0
</code></pre>
<hr />
<p>As pointed in below link, <code>kubelet</code> can automatically renew it's certificate (<code>kubeadm</code> configures the cluster in a way that this option is enabled):</p>
<ul>
<li><em><a href="https://kubernetes.io/docs/tasks/tls/certificate-rotation/" rel="noreferrer">Kubernetes.io: Configure Certificate Rotation for the Kubelet</a></em></li>
<li><em><a href="https://github.com/kubernetes/kubeadm/issues/2185#issuecomment-644260417" rel="noreferrer">Github.com: Kubernetes: Kubeadm: Issues: --certificate-renewal true doesn't renew kubelet.conf</a></em></li>
</ul>
<p>Depending on the version used in your environment, this can be disabled. Currently in the newest version of k8s managed by <code>kubeadm</code> this option is enabled by default according to my knowledge.</p>
<hr />
<p>Please keep in mind that before you start with any kubernetes node/control plane/update/upgrade to <strong>read &quot;Urgent Upgrade Notes&quot; specific to your k8s version</strong> (example):</p>
<ul>
<li><a href="https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.21.md#urgent-upgrade-notes" rel="noreferrer">Github.com: Kubernetes: CHANGELOG: 1.21: Urgent upgrade nodes</a></li>
</ul>
<hr />
<p>Defining the automatic way of certificate rotation could go in either way but you can use already mentioned commands to automate this process. You would need to create a script (which you already have) that would be put in cron that would fire after some time and renew them.</p>
