<p>Currently running a k8s cluster however occasionally I get memory issues. The following error will pop up, </p>

<p><code>Failed create pod sandbox: rpc error: code = Unknown desc = failed to create a sandbox for pod "&lt;web app&gt;": Error response from daemon: devmapper: Thin Pool has 6500 free data blocks which is less than minimum required 7781 free data blocks. Create more free space in thin pool or use dm.min_free_space option to change behavior</code></p>

<p>I can resolve this by manually running <code>docker ps -a -f status=exited -q | xargs -r docker rm -v</code></p>

<p>However I want Kubernetes to do this work itself. Currently in my kublet config I have:</p>

<pre><code>evictionHard:
  imagefs.available: 15%
  memory.available: "100Mi"
  nodefs.available: 10%
  nodefs.inodesFree: 5%
imageGCHighThresholdPercent: 85
imageGCLowThresholdPercent: 80
</code></pre>

<p>What am i doing wrong?</p>

D_G
<p>Reading the error you've posted seems to me you are using "<a href="https://docs.docker.com/storage/storagedriver/device-mapper-driver/" rel="nofollow noreferrer">devicemapper</a>" as storage driver.</p>

<blockquote>
  <p>The <code>devicemapper</code> storage driver is deprecated in Docker Engine 18.09, and will be removed in a future release. It is recommended that users of the <code>devicemapper</code> storage driver migrate to <code>overlay2</code>. </p>
</blockquote>

<p>I should suggest you use "<strong>overlay2</strong>" as storage drive, unless you are running a non-support OS. See <a href="https://docs.docker.com/v17.09/engine/userguide/storagedriver/selectadriver/#docker-ce" rel="nofollow noreferrer">here</a> the support OS versions.</p>

<p>You can check your actual storage drive using <code>docker info</code> command, you will get an output like this:</p>

<pre><code>Client:
 Debug Mode: false

Server:
 Containers: 21
  Running: 18
  Paused: 0
  Stopped: 3
 Images: 11
 Server Version: 19.03.5
 Storage Driver: devicemapper &lt;&lt;== See here
  Pool Name: docker-8:1-7999625-pool
  Pool Blocksize: 65.54kB
...
</code></pre>

<p>>
Supposing you want to change the storage drive from <code>devicemapper</code> to <code>overlay2</code>, you need to following this steps:</p>

<p><strong><em>Changing the storage driver makes existing containers and images inaccessible on the local system. Use  <code>docker save</code>  to save any images you have built or push them to Docker Hub or a private registry before changing the storage driver, so that you do not need to re-create them later.</em></strong></p>

<p>Before following this procedure, you must first meet all the <a href="https://docs.docker.com/storage/storagedriver/overlayfs-driver/#prerequisites" rel="nofollow noreferrer">prerequisites</a>.</p>

<ol>
<li><p>Stop Docker.</p>

<pre><code>$ sudo systemctl stop docker
</code></pre></li>
<li><p>Copy the contents of  <code>/var/lib/docker</code>  to a temporary location.</p>

<pre><code>$ cp -au /var/lib/docker /var/lib/docker.bk
</code></pre></li>
<li><p>If you want to use a separate backing filesystem from the one used by  <code>/var/lib/</code>, format the filesystem and mount it into  <code>/var/lib/docker</code>. Make sure add this mount to  <code>/etc/fstab</code>  to make it permanent.</p></li>
<li><p>Edit  <code>/etc/docker/daemon.json</code>. If it does not yet exist, create it. Assuming that the file was empty, add the following contents.</p>

<pre><code>{
  "storage-driver": "overlay2"
}
</code></pre>

<p>Docker does not start if the  <code>daemon.json</code>  file contains badly-formed JSON.</p></li>
<li><p>Start Docker.</p>

<pre><code>$ sudo systemctl start docker
</code></pre></li>
<li><p>Verify that the daemon is using the  <code>overlay2</code>  storage driver. Use the  <code>docker info</code>  command and look for  <code>Storage Driver</code>  and  <code>Backing filesystem</code>.</p></li>
</ol>

<pre><code>Client:
 Debug Mode: false

Server:
 Containers: 35
  Running: 15
  Paused: 0
  Stopped: 20
 Images: 11
 Server Version: 19.03.5
 Storage Driver: overlay2 &lt;=== HERE
  Backing Filesystem: extfs &lt;== HERE
  Supports d_type: true
</code></pre>

<p>Extracted from <a href="https://docs.docker.com/storage/storagedriver/overlayfs-driver/" rel="nofollow noreferrer">Docker Documentation.</a></p>
