<p>I read carefully the Kubernetes Documentation <a href="https://kubernetes.io/docs/tasks/administer-cluster/out-of-resource/" rel="nofollow noreferrer">here</a> about extending the default 15% of <code>imagefs.available</code> and the others parameters but it doesn't say how to set it, i have installed the RKE (Rancher Kubernetes Engine) with the following configs.</p>
<pre><code>Kubernets Version:v1.17.5
OS: Ubuntu 18.04.4 LTS
Container Runtime: 19.3.9
</code></pre>
<p>Btw the documentation doesn't clearly explain how to extend the default eviction so i come here to know if someone can tell me how to extend it .</p>
<p>I also tried kubelet command line but i get this</p>
<pre><code>I0903 09:59:13.143727   31472 plugins.go:100] No cloud provider specified.
W0903 09:59:13.143760   31472 server.go:560] standalone mode, no API client
W0903 09:59:13.148508   31472 container_manager_linux.go:912] CPUAccounting not enabled for pid: 31472
W0903 09:59:13.148527   31472 container_manager_linux.go:915] MemoryAccounting not enabled for pid: 31472
W0903 09:59:13.579594   31472 server.go:474] No api server defined - no events will be sent to API server.
I0903 09:59:13.579620   31472 server.go:647] --cgroups-per-qos enabled, but --cgroup-root was not specified.  defaulting to /
F0903 09:59:13.581308   31472 server.go:274] failed to run Kubelet: running with swap on is not supported, please disable swap! or set --fail-swap-on flag to false. /proc/swaps contained: [Filename                           Type            Size    Used    Priority /dev/md0                                partition      16758780       9846712  -2]
</code></pre>
<p>Additional Information
Server(Cloud,Bare metal,etc..): Bare Metal
Kubernet installed using: rancher</p>
<p>when i do it by config.yaml with the kuberconfig file provided by rancher it try to do eviction of my containers in docker and not in kubernets containers.</p>
<p>So my command is something like this
kubelet --kubeconfig=/root/.kube/config --config=/root/.kube/config.yaml</p>
<pre><code>E0903 12:29:25.934582   26731 remote_image.go:135] RemoveImage &quot;sha256:8c8e0533fa439319ae0ada18d3a7380d33f14f313ea5dd750ab68a1fde65d909&quot; from image service failed: rpc error: code = Unknown desc = Error response from daemon: conflict: unable to remove repository reference &quot;rancher/rke-tools:v0.1.56&quot; (must force) - container 2e31fd3c915d is using its referenced image 8c8e0533fa43
E0903 12:29:25.934602   26731 kuberuntime_image.go:120] Remove image &quot;sha256:8c8e0533fa439319ae0ada18d3a7380d33f14f313ea5dd750ab68a1fde65d909&quot; failed: rpc error: code = Unknown desc = Error response from daemon: conflict: unable to remove repository reference &quot;rancher/rke-tools:v0.1.56&quot; (must force) - container 2e31fd3c915d is using its referenced image 8c8e0533fa43
I0903 12:29:25.934614   26731 image_gc_manager.go:376] [imageGCManager]: Removing image &quot;sha256:2b5329557072f7cd9591670db3814bb54a72aa474ca301f00c8a29597cd32c94&quot; to free 18234790 bytes
E0903 12:29:25.935726   26731 remote_image.go:135] RemoveImage &quot;sha256:2b5329557072f7cd9591670db3814bb54a72aa474ca301f00c8a29597cd32c94&quot; from image service failed: rpc error: code = Unknown desc = Error response from daemon: conflict: unable to remove repository reference &quot;willfarrell/autoheal:latest&quot; (must force) - container 5fb28a06d30d is using its referenced image 2b5329557072
E0903 12:29:25.935746   26731 kuberuntime_image.go:120] Remove image &quot;sha256:2b5329557072f7cd9591670db3814bb54a72aa474ca301f00c8a29597cd32c94&quot; failed: rpc error: code = Unknown desc = Error response from daemon: conflict: unable to remove repository reference &quot;willfarrell/autoheal:latest&quot; (must force) - container 5fb28a06d30d is using its referenced image 2b5329557072
I0903 12:29:25.935758   26731 image_gc_manager.go:376] [imageGCManager]: Removing image &quot;sha256:0a3db21bc2b49b5cddea7ead8b99387ab37a332b71f0b016e1983587bb7d24af&quot; to free 421267928 bytes
E0903 12:29:25.937276   26731 remote_image.go:135] RemoveImage &quot;sha256:0a3db21bc2b49b5cddea7ead8b99387ab37a332b71f0b016e1983587bb7d24af&quot; from image service failed: rpc error: code = Unknown desc = Error response from daemon: conflict: unable to remove repository reference &quot;linuxserver/plex:latest&quot; (must force) - container 8ca7a3b6202b is using its referenced image 0a3db21bc2b4
E0903 12:29:25.937295   26731 kuberuntime_image.go:120] Remove image &quot;sha256:0a3db21bc2b49b5cddea7ead8b99387ab37a332b71f0b016e1983587bb7d24af&quot; failed: rpc error: code = Unknown desc = Error response from daemon: conflict: unable to remove repository reference &quot;linuxserver/plex:latest&quot; (must force) - container 8ca7a3b6202b is using its referenced image 0a3db21bc2b4
I0903 12:29:25.937307   26731 image_gc_manager.go:376] [imageGCManager]: Removing image &quot;sha256:7aab9b2c0cd6d39e03815bf3429be845257a016c3a3f5fd9e22b1ec5c3e9f329&quot; to free 291526564 bytes
E0903 12:29:25.938673   26731 remote_image.go:135] RemoveImage &quot;sha256:7aab9b2c0cd6d39e03815bf3429be845257a016c3a3f5fd9e22b1ec5c3e9f329&quot; from image service failed: rpc error: code = Unknown desc = Error response from daemon: conflict: unable to remove repository reference &quot;rancher/rancher-agent:v2.4.3&quot; (must force) - container 9eb5c8663b7d is using its referenced image 7aab9b2c0cd6
E0903 12:29:25.938693   26731 kuberuntime_image.go:120] Remove image &quot;sha256:7aab9b2c0cd6d39e03815bf3429be845257a016c3a3f5fd9e22b1ec5c3e9f329&quot; failed: rpc error: code = Unknown desc = Error response from daemon: conflict: unable to remove repository reference &quot;rancher/rancher-agent:v2.4.3&quot; (must force) - container 9eb5c8663b7d is using its referenced image 7aab9b2c0cd6
W0903 12:29:25.938733   26731 eviction_manager.go:430] eviction manager: unexpected error when attempting to reduce inodes pressure: wanted to free 9223372036854775807 bytes, but freed 0 bytes space with errors in image deletion: [rpc error: code = Unknown desc = Error response from daemon: conflict: unable to remove repository reference &quot;portainer/portainer:latest&quot; (must force) - container 6193548811df is using its referenced image 2869fc110bf7, rpc error: code = Unknown desc = Error response from daemon: conflict: unable to remove repository reference &quot;rancher/coreos-etcd:v3.4.3-rancher1&quot; (must force) - container fd5c341853c0 is using its referenced image a0b920cf970d, rpc error: code = Unknown desc = Error response from daemon: conflict: unable to remove repository reference &quot;rancher/hyperkube:v1.17.5-rancher1&quot; (must force) - container 640e1d162e1e is using its referenced image ff99d966b0ee, rpc error: code = Unknown desc = Error response from daemon: conflict: unable to remove repository reference &quot;hawkinzzz/dngele:latest&quot; (must force) - container c16369f9385b is using its referenced image f3eb38637bfa, rpc error: code = Unknown desc = Error response from daemon: conflict: unable to remove repository reference &quot;pyouroboros/ouroboros:latest&quot; (must force) - container c0e2abfb3104 is using its referenced image 6bde18e83f41, rpc error: code = Unknown desc = Error response from daemon: conflict: unable to remove repository reference &quot;linuxserver/emby:beta&quot; (must force) - container 3def488e8543 is using its referenced image 95d10324f024, rpc error: code = Unknown desc = Error response from daemon: conflict: unable to remove repository reference &quot;physk/rclone-mergerfs:latest&quot; (must force) - container 9affc85735a1 is using its referenced image 80e6bae8490b, rpc error: code = Unknown desc = Error response from daemon: conflict: unable to remove repository reference &quot;rancher/rancher:latest&quot; (must force) - container 549b67e38726 is using its referenced image ddb8c914dee4, rpc error: code = Unknown desc = Error response from daemon: conflict: unable to remove repository reference &quot;rancher/rke-tools:v0.1.56&quot; (must force) - container 2e31fd3c915d is using its referenced image 8c8e0533fa43, rpc error: code = Unknown desc = Error response from daemon: conflict: unable to remove repository reference &quot;willfarrell/autoheal:latest&quot; (must force) - container 5fb28a06d30d is using its referenced image 2b5329557072, rpc error: code = Unknown desc = Error response from daemon: conflict: unable to remove repository reference &quot;linuxserver/plex:latest&quot; (must force) - container 8ca7a3b6202b is using its referenced image 0a3db21bc2b4, rpc error: code = Unknown desc = Error response from daemon: conflict: unable to remove repository reference &quot;rancher/rancher-agent:v2.4.3&quot; (must force) - container 9eb5c8663b7d is using its referenced image 7aab9b2c0cd6]
I0903 12:29:25.951254   26731 eviction_manager.go:355] eviction manager: must evict pod(s) to reclaim inodes
I0903 12:29:25.951274   26731 eviction_manager.go:373] eviction manager: pods ranked for eviction: canal-l29wx_kube-system(421796dc-24b3-4bcb-97f3-f61e36665c05)
E0903 12:29:25.951289   26731 eviction_manager.go:575] eviction manager: cannot evict a critical pod canal-l29wx_kube-system(421796dc-24b3-4bcb-97f3-f61e36665c05)
I0903 12:29:25.951297   26731 eviction_manager.go:396] eviction manager: unable to evict any pods from the node

</code></pre>
<p>Kind regards</p>

David Martins
<blockquote>
<p>The kubelet has the following default hard eviction threshold:
memory.available&lt;100Mi nodefs.available&lt;10% nodefs.inodesFree&lt;5%
imagefs.available&lt;15%</p>
</blockquote>
<p>As per <a href="https://rancher.com/docs/rancher/v2.x/en/faq/technical/#how-can-i-add-additional-arguments-binds-environment-variables-to-kubernetes-components-in-a-rancher-launched-kubernetes-cluster" rel="nofollow noreferrer">official Rancher page</a>:</p>
<p>You can add additional arguments/binds/environment variables via the <a href="https://rancher.com/docs/rancher/v2.x/en/cluster-provisioning/rke-clusters/options/#config-file" rel="nofollow noreferrer">Config File option</a> in Cluster Options. For more information, see the <a href="https://rancher.com/docs/rke/latest/en/config-options/services/services-extras/" rel="nofollow noreferrer">Extra Args, Extra Binds, and Extra Environment Variables</a> in the RKE documentation or browse the <a href="https://rancher.com/docs/rke/latest/en/example-yamls/" rel="nofollow noreferrer">Example Cluster.ymls</a>.</p>
<p>Look in the <a href="https://rancher.com/docs/rke/latest/en/example-yamls/#full-cluster-yml-example" rel="nofollow noreferrer">full example</a> how you can configure kubelet options:</p>
<pre><code>kubelet:
      # Base domain for the cluster
      cluster_domain: cluster.local
      # IP address for the DNS service endpoint
      cluster_dns_server: 10.43.0.10
      # Fail if swap is on
      fail_swap_on: false
      # Set max pods to 250 instead of default 110
      extra_args:
        max-pods: 250
</code></pre>
<p>Here you can add smth like below or whatever you exactly need.</p>
<pre><code>eviction-hard: &quot;nodefs.available&lt;1%,imagefs.available&lt;1%&quot;
eviction-soft-grace-period: &quot;nodefs.available=30s,imagefs.available=30s&quot;
eviction-soft: &quot;nodefs.available&lt;1%,imagefs.available&lt;1%&quot;
</code></pre>
<p>Frankly speaking im not sure if you can change RKE's kubelet on flight, but for sure you can create cluster from scratch with required options.</p>
