<p>We have a pod in a Google Cloud Platform Kubernetes cluster writing JsonFormatted to StdOut. This is picked up by Stackdriver out of box. However, we see the disk usage of the pod just growing and growing, and we can't understand how to set a max size on the Deployment for log rotation.</p>

<p>Documentation on Google Cloud and Kubernetes is unclear on this.</p>

<p>This is just the last hour:</p>

<p><a href="https://i.stack.imgur.com/xbwqg.png" rel="noreferrer"><img src="https://i.stack.imgur.com/xbwqg.png" alt="Memory consumption on a Pod"></a></p>

chrisva
<p>Are you sure that disk usage of the pod is high because of the logs?
If the application writes logs to stdout, it doesn't use any disk space inside the pod.
All logs are usually stored in a log file on the node’s filesystem and can be managed by the node logrotate process.</p>
<p>Perhaps application uses pod's disk space for something else, like temp files or debug information?</p>
<p>Here is the part of <a href="https://kubernetes.io/docs/concepts/cluster-administration/logging/" rel="nofollow noreferrer">documentation</a> related to log rotation:</p>
<blockquote>
<h2>Logging at the node level:</h2>
<p>Everything a containerized application writes to stdout and stderr is
handled and redirected somewhere by a container engine. For example,
the Docker container engine redirects those two streams to a <a href="https://docs.docker.com/engine/admin/logging/overview" rel="nofollow noreferrer">logging
driver</a>, which is configured in Kubernetes to write to a file in
json format.</p>
<p>An important consideration in node-level logging is implementing log
rotation, so that logs don’t consume all available storage <strong>on the
node</strong>.</p>
<p>Kubernetes currently is not responsible for rotating logs, but rather
a deployment tool should set up a solution to address that. For
example, in Kubernetes clusters, deployed by the kube-up.sh script,
there is a logrotate tool configured to run each hour.</p>
<p>You can also set up a container runtime to rotate application’s logs
automatically, e.g. by using Docker’s log-opt.</p>
<p>In the kube-up.sh script, the latter approach is used for COS image on
GCP, and the former approach is used in any other environment.   In
both cases, by default rotation is configured to take place when log
file exceeds 10MB.</p>
<p>As an example, you can find detailed information about how kube-up.sh
sets up logging for COS image on GCP in the corresponding <a href="https://github.com/kubernetes/kubernetes/blob/master/cluster/gce/gci/configure-helper.sh#L324" rel="nofollow noreferrer">script</a>.</p>
</blockquote>
<p>Here is a part of the script related to logrotate:</p>
<pre><code># Installs logrotate configuration files
function setup-logrotate() {
  mkdir -p /etc/logrotate.d/
  # Configure log rotation for all logs in /var/log, which is where k8s services
  # are configured to write their log files. Whenever logrotate is ran, this
  # config will:
  # * rotate the log file if its size is &gt; 100Mb OR if one day has elapsed
  # * save rotated logs into a gzipped timestamped backup
  # * log file timestamp (controlled by 'dateformat') includes seconds too. This
  #   ensures that logrotate can generate unique logfiles during each rotation
  #   (otherwise it skips rotation if 'maxsize' is reached multiple times in a
  #   day).
  # * keep only 5 old (rotated) logs, and will discard older logs.
  cat &gt; /etc/logrotate.d/allvarlogs &lt;&lt;EOF
/var/log/*.log {
    rotate ${LOGROTATE_FILES_MAX_COUNT:-5}
    copytruncate
    missingok
    notifempty
    compress
    maxsize ${LOGROTATE_MAX_SIZE:-100M}
    daily
    dateext
    dateformat -%Y%m%d-%s
    create 0644 root root
}
EOF

}
</code></pre>
