<p>I am maintaining a K8s cluster and recently kubelet evicted pods many times on different nodes because of disk pressure. After investigation, I found out that the problem is the container log files at <code>/var/lib/docker/containers/.../*-json.log</code> and these files can grow to hundreds of Gi and consume all of the disk.</p>

<p>I even face this when I was using a central logging stack consists of Kibana, Elasticsearch and Fluentbit. The fluentbit logs were around 500 Gi and after removing the central logging stack the disk pressure almost solved. But now I see it for some of my other components and its logs are consuming around 170 Gi.</p>

<p>What is some of the best practices and tools for managing log files in k8s?</p>

<p>Every Kubernetes installer should include Logrotate to handle this. <a href="https://kubernetes.io/docs/concepts/cluster-administration/logging/" rel="nofollow noreferrer">https://kubernetes.io/docs/concepts/cluster-administration/logging/</a> has some basic info but it depends on your exact configuration.</p>

<p>EDIT: As I have now informed myself, Docker itself can also do log rotation directly so that's an option too.</p>
