<p>we have a kubernetes cluster running on Centos 7. However all logging is going to /var/log/messages which is making centos system logs hard to read. Is there a way I can tell kubeadm/kubernetes to log to /var/log/kubernetes rather?</p>

<p>We are already sending our application (pod) logs to a mountpoint. We need to move the stderr logs of kubernetes.</p>

<blockquote>
  <p>However all logging is going to /var/log/messages which is making centos system logs hard to read. Is there a way I can tell kubeadm/kubernetes to log to /var/log/kubernetes rather?</p>
</blockquote>

<p>No, not exactly, but you can reconfigure Docker to log in a different way.</p>

<p>This might depend on the Docker version you're running but in my CentOS 7 VM (a couple of weeks old) i'm running Docker version <code>1.13.1</code>, installed via <code>yum</code>. </p>

<p>When looking through the <a href="https://docs.docker.com/v1.13/engine/admin/logging/overview/" rel="nofollow noreferrer">docs</a> for version <code>1.13</code> and the latest <a href="https://docs.docker.com/config/containers/logging/configure/" rel="nofollow noreferrer">stable</a> version of Docker they say more or less the same thing:</p>

<blockquote>
  <p>If you do not specify a logging driver, the default is json-file. </p>
</blockquote>

<p>The version of Docker i installed via <code>yum</code> had the following line in an environment file (<code>/etc/sysconfig/docker</code>) that is loaded when starting Docker:</p>

<pre><code>OPTIONS='--selinux-enabled --log-driver=journald --signature-verification=false'
</code></pre>

<p>As you can see the logging driver are configured as <code>journald</code>, that should be the reason you're seeing logs from your containers in <code>/var/log/messages</code>. You can check which logging drive are configured with:</p>

<pre><code>docker info | grep 'Logging Driver'
</code></pre>

<p>The logging driver decides where all of the logs, in Docker meaning <code>stderr</code> and <code>stdout</code> from containers are sent. Docker supports a couple of different <a href="https://docs.docker.com/config/containers/logging/configure/#supported-logging-drivers" rel="nofollow noreferrer">logging drivers</a>, if you choose to configure e.g. <code>json-file</code> which might be the best choice if you want to relocate the logging from an OS perspective ("changing" the log path). Every Docker container will have it's own log written to <code>/var/log/pods/&lt;ID&gt;/&lt;NAME&gt;/&lt;LOGFILE&gt;</code>, actually the log files are symlinks back to <code>/var/lib/docker/containers/&lt;ID&gt;/&lt;ID&gt;-json.log</code>.</p>

<p>If you do configure <code>json-file</code> then remove the <code>--log-driver=journald</code> flag and instead configure this in the <code>/etc/docker/daemon.json</code> file, mentioned in the docs. With <code>json-file</code> you can configure things like log rotation and log file sizes, please consult the <a href="https://docs.docker.com/config/containers/logging/json-file/#options" rel="nofollow noreferrer">docs</a> for more options.</p>

<p>When configuring via the <code>daemon.json</code> file this becomes a global setting, you can always override the logging driver used for a specific container with <code>docker run ... --log-driver</code>.</p>

<p>These logging changes applies for everything running within Docker, to move logging for e.g. <code>kubelet</code> which runs alongside Docker on your host you can look at the configurable <a href="https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/#options" rel="nofollow noreferrer">options</a>. Default the <code>kubelet</code> <code>stderr</code> logs are logged via <code>journald</code> and ends up in <code>/var/log/messages</code>, to change this behavior can add the <code>--log-dir</code> options and point to another location.</p>

<p>In the end of the day i think it's good to give log shipping a thought and investigate the other logging drivers if they might fit into your environment.</p>
