<p>I have a Kubernetes-cluster with 1 master-node and 3 worker-nodes. All Nodes are running on CentOS 7 with Docker 19.06. I'm also running Longhorn for dynamic provisioning of volumes (if that's important).</p>
<p>My problem is that every few days one of the worker nodes grows the HDD-usage to 85% (43GB). This is not a linear increase, but happens over a few hours, sometimes quite rapidly. I can &quot;solve&quot; this problem for a few days by first restarting the docker service and then doing a <code>docker system prune -a</code>. If I don't restart the service first, the prune removes next to nothing (only a few MB).</p>
<p>I also tried to find out which container is taking up all that space, but <code>docker system df</code> says it doesn't use the space. I used <code>df</code> and <code>du</code> to crawl along the /var/lib/docker subdirectories too, and it seems none of the folders (alone or all together) takes up much space either. Continuing this all over the system, I can't find any other big directories either. There are 24GB that I just can't account for. What makes me think this is a docker problem nonetheless is that a restart and prune just solves it every time.</p>
<p>Googling around I found a lot of similar issues where most people just decided to increase disk space. I'm not keen on accepting this as the preferred solution, as it feels like kicking the can down the road.</p>
<p>Would you have any smart ideas on what to do instead of increasing disk space?</p>

<p>It seems like it is expected behavior, from <a href="https://docs.docker.com/config/pruning/" rel="nofollow noreferrer">Docker documentation</a> you can read:</p>
<blockquote>
<p>Docker takes a conservative approach to cleaning up unused objects
(often referred to as “garbage collection”), such as images,
containers, volumes, and networks: these objects are generally not
removed unless you explicitly ask Docker to do so. This can cause
Docker to use extra disk space. For each type of object, Docker
provides a <code>prune</code> command. In addition, you can use <code>docker system prune</code> to clean up multiple types of objects at once. This topic shows
how to use these <code>prune</code> commands.</p>
</blockquote>
<p>So it seems like you have to clean it up manually using <code>docker system/image/container prune</code>.  Other issue might be that those containers create too much logs and you might need to <a href="https://stackoverflow.com/questions/42510002/how-to-clear-the-logs-properly-for-a-docker-container">clean it up</a>.</p>
