<p>When I upgrade kubernetes version 1.20.X to 1.21.1, all containers related is up to date. But the pause container is still in use, I can not force update it to the latest version.</p>
<pre><code># docker ps
XXX/pause:3.2
# docker images
XXX/pause:3.2
XXX/pause:3.4.1
# docker rmi -f XXX/pause:3.2
Error response from daemon: conflict: unable to delete XXX/pause:3.2 (cannot be forced) - image is being used by running container
</code></pre>

<p>When you upgrade the cluster using <code>kubeadm</code> you will probably get the notification about the <code>kubelet</code> manual upgrade requirement:</p>
<pre><code>Components that must be upgraded manually after you have upgraded the control plane with 'kubeadm upgrade apply':
COMPONENT   CURRENT       TARGET
kubelet     1 x v1.20.7   v1.21.1
</code></pre>
<p>I've managed to create a <code>kubeadm</code> cluster version: <code>1.20.7-00</code> and then upgraded the cluster to the newest available at the time: <code>1.21.1-00</code>. After the upgrade was complete, the pause container stayed in the version <code>3.2.0</code> even after upgrading <code>kubelet</code>.</p>
<p><strong>One of the ways</strong> to update <code>kubelet</code> to use specific <code>pause</code> container version is by:</p>
<ul>
<li>modifiying following file:
<ul>
<li><code>/var/lib/kubelet/kubeadm-flags.env</code> (change for example to <code>k8s.gcr.io/pause:3.3</code>)</li>
</ul>
</li>
</ul>
<pre class="lang-sh prettyprint-override"><code>KUBELET_KUBEADM_ARGS=&quot;--network-plugin=cni --pod-infra-container-image=k8s.gcr.io/pause:3.2&quot;
</code></pre>
<ul>
<li>restarting kubelet (depending on the OS)
<ul>
<li><code>$ systemctl restart kubelet</code></li>
</ul>
</li>
</ul>
<p>After this steps you should be seeing the new version of <code>pause</code> container passed to <code>kubelet</code>.</p>
<ul>
<li><code>$ systemctl status kubelet</code></li>
</ul>
<pre class="lang-sh prettyprint-override"><code>
kruk@ubuntu:~$ systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
     Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/kubelet.service.d
             └─10-kubeadm.conf
     Active: active (running) since Thu 2021-05-27 13:28:12 UTC; 7h ago
       Docs: https://kubernetes.io/docs/home/
   Main PID: 724 (kubelet)
      Tasks: 18 (limit: 9442)
     Memory: 128.6M
     CGroup: /system.slice/kubelet.service
             └─724 /usr/bin/kubelet &lt;-SKIPPED-&gt; --pod-infra-container-image=k8s.gcr.io/pause:3.3

May 27 13:29:12 ubuntu kubelet[724]: 2021-05-27 13:29:12.125 [INFO][5164] ipam.go 1068: Successfully claimed IPs: [172.16.243.205/26] block=172.16.243.192/26 handle=&quot;k8s-pod-network.1638a3ba44d1a46f6ad7eadb1519a42cdda98fafd0c94a7b67881f38213a5032&quot; host=&quot;ubuntu&quot;
May 27 13:29:12 ubuntu kubelet[724]: 2021-05-27 13:29:12.125 [INFO][5164] ipam.go 722: Auto-assigned 1 out of 1 IPv4s: [172.16.243.205/26] handle=&quot;k8s-pod-network.1638a3ba44d1a46f6ad7eadb1519a42cdda98fafd0c94a7b67881f38213a5032&quot; host=&quot;ubuntu&quot;
May 27 13:29:12 ubuntu kubelet[724]: time=&quot;2021-05-27T13:29:12Z&quot; level=info msg=&quot;Released host-wide IPAM lock.&quot; source=&quot;ipam_plugin.go:369&quot;
</code></pre>
<p>In my testing the old container that were present were not updated to the new <code>pause</code> container. They stayed at version <code>3.2</code>. Each new workload that was spawned, like for example <code>nginx</code> <code>Deployment</code> was using new <code>pause</code> container version:</p>
<ul>
<li><code>$ docker ps</code></li>
</ul>
<pre><code>CONTAINER ID   IMAGE                  COMMAND                  CREATED       STATUS       PORTS     NAMES
1cc215019335   nginx                  &quot;/docker-entrypoint.…&quot;   7 hours ago   Up 8 hours             k8s_nginx_nginx-6799fc88d8-lhh48_default_58580cf2-ac6c-4d55-9c08-608ce2018fce_1
1638a3ba44d1   k8s.gcr.io/pause:3.3   &quot;/pause&quot;                 7 hours ago   Up 8 hours             k8s_POD_nginx-6799fc88d8-lhh48_default_58580cf2-ac6c-4d55-9c08-608ce2018fce_1
</code></pre>
<hr />
<p>Additional resources/reference on the topic:</p>
<ul>
<li><em><a href="https://www.ianlewis.org/en/almighty-pause-container" rel="nofollow noreferrer">Ianlewis.org: Almighty pause container</a></em></li>
<li><em><a href="https://github.com/kubernetes/kubernetes/issues/98765" rel="nofollow noreferrer">Github.com: Kubernetes: Isuses: Handling Deprecation of pod-infra-container-image</a></em></li>
<li><em><a href="https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/kubelet-integration/#workflow-when-using-kubeadm-init" rel="nofollow noreferrer">Kubernetes.io: Docs: Setup: Production environment: Tools: Kubeadm: Kubelet integration: Workflow when using kubeadm init</a></em></li>
</ul>
