<p>I am facing a problem with kubernetes nodes deployed on AWS.
(Cluster with 3 nodes and 1 master running on m3.large instances with each about 25GB)</p>

<p>After (about 3 days) there is 0KB left on disk and the cluster get stuck.</p>

<p>All the storage (more or less) is used by /var/lib/docker/overlay/.
Inside this folder are about 500 or more of those files:</p>

<pre><code>drwx------ 3 root root 4096 Jun 20 15:33 ed4f90bd7a64806f9917e995a02974ac69883a06933033ffd5049dd31c13427a
drwx------ 3 root root 4096 Jun 20 15:28 ee9344fea422c38d71fdd2793ed517c5693b0c8a890964e6932befa1ebe5aa63
drwx------ 3 root root 4096 Jun 20 16:17 efed310a549243e730e9796e558b2ae282e07ea3ce0840a50c0917a435893d42
drwx------ 3 root root 4096 Jun 20 14:39 eff7f04f17c0f96cff496734fdc1903758af1dfdcd46011f6c3362c73c6086c2
drwx------ 3 root root 4096 Jun 20 15:29 f5bfb696f5a6cad888f7042d01bfe146c0621517c124d58d76e77683efa1034e
drwx------ 3 root root 4096 Jun 20 15:26 f5fa9d5d2066c7fc1c8f80970669634886dcaccc9e73ada33c7c250845d2fe8c
drwx------ 3 root root 4096 Jun 20 14:38 f8fd64fb1e0ab26708d5458dddd2d5a70018034237dfed3db48ada5666fcf77f
drwx------ 3 root root 4096 Jun 20 14:46 faa143ebd7a4079eaa45ddbf17dcfc9163e3035983f2e334e32a60e89452fa94
drwx------ 3 root root 4096 Jun 20 14:48 fb93c0c64e0d4935bf67fc8f70df2b8a4cffe59e294ee8a876dfdf6b57486da5
drwx------ 3 root root 4096 Jun 20 14:46 fd0a420d5655fb7d022c397effdb95968ff7e722c58fcc7915f97e8df47cd080
</code></pre>

<p>Cluster runs on Kubernetes 1.6.4 and Docker 1.12.6.</p>

<p>Seems to be a problem with the garbage collector of kubernetes. Running cAdvisor /validate gives me following message</p>

<pre><code> None of the devices support 'cfq' I/O scheduler. No disk stats can be reported.
     Disk "xvda" Scheduler type "none".
</code></pre>

<p>Running those statement <code>journalctl -u kubelet | grep -i garbage</code> gives also a error message:
<code>Jun 20 14:35:21 ip-172-21-4-239 kubelet[1551]: E0620 14:35:21.986898    1551 kubelet.go:1165] Image garbage collection failed: unable to find data for container /</code></p>

<p>Any ideas how to get the garbage collector working again?</p>

<p>I was able to resolve a similar issue with recurring high IO on the nodes due to <code>du -s /var/lib/docker/overlay/</code> by editing <code>kops cluster.spec</code> with <code>kops edit cluster [cluster_name]</code>.
I added the following under specs:</p>

<pre><code>docker:
    logDriver: json-file
    logLevel: warn
    storage: overlay2
</code></pre>

<p>It looks like by default <code>kops</code> configures docker to use overlay as default storage driver, while docker recommends using <code>overlay2</code> as newer, more stable and faster.</p>
