<p>A node of my k8s cluster has GC trying to remove images used by a container. </p>

<p>This behaviour seems strange to me.</p>

<p>Here the logs:</p>

<pre><code>kubelet: I1218 12:44:19.925831   11177 image_gc_manager.go:334] [imageGCManager]: Removing image "sha256:99e59f495ffaa222bfeb67580213e8c28c1e885f1d245ab2bbe3b1b1ec3bd0b2" to free 746888 bytes
kubelet: E1218 12:44:19.928742   11177 remote_image.go:130] RemoveImage "sha256:99e59f495ffaa222bfeb67580213e8c28c1e885f1d245ab2bbe3b1b1ec3bd0b2" from image service failed: rpc error: code = Unknown desc = Error response from daemon: conflict: unable to delete 99e59f495ffa (cannot be forced) - image is being used by running container 6f236a385a8e
kubelet: E1218 12:44:19.928793   11177 kuberuntime_image.go:126] Remove image "sha256:99e59f495ffaa222bfeb67580213e8c28c1e885f1d245ab2bbe3b1b1ec3bd0b2" failed: rpc error: code = Unknown desc = Error response from daemon: conflict: unable to delete 99e59f495ffa (cannot be forced) - image is being used by running container 6f236a385a8e
kubelet: W1218 12:44:19.928821   11177 eviction_manager.go:435] eviction manager: unexpected error when attempting to reduce nodefs pressure: wanted to free 9223372036854775807 bytes, but freed 0 bytes space with errors in image deletion: rpc error: code = Unknown desc = Error response from daemon: conflict: unable to delete 99e59f495ffa (cannot be forced) - image is being used by running container 6f236a385a8e
</code></pre>

<p>Any suggestions?
May a manual remove of docker images and stopped containers on a node cause such a problem?</p>

<p>Thank you in advance.</p>

<p>What you've encountered is not the regular Kubernetes garbage collection that deleted orphaned API resource objects, but the <a href="https://kubernetes.io/docs/concepts/cluster-administration/kubelet-garbage-collection/#image-collection" rel="nofollow noreferrer">kubelet's <em>Image collection</em></a>.</p>

<p>Whenever a node experiences <em>Disk pressure</em>, the Kubelet daemon will desperately try to reclaim disk space by deleting (supposedly) unused images. Reading the <a href="https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/images/image_gc_manager.go" rel="nofollow noreferrer">source code</a> shows that the Kubelet sorts the images to remove by the time since they have last been used for creating a Pod -- if all images are in use, the Kubelet will try to delete them anyways and fail (which is probably what happened to you).</p>

<p>You can use the Kubelet's <code>--minimum-image-ttl-duration</code> flag to specify a minimum age that an image needs to have before the Kubelet will ever try to remove it (although this will not prevent the Kubelet from trying to remove used images altogether). Alternatively, see if you can provision your nodes with more disk space for images (or build smaller images).</p>
