<p>How to force all kubernetes services (proxy, kublet, apiserver..., containers) to write logs to <code>/var/logs</code>?</p>
<p>For example:</p>
<pre><code>/var/logs/apiServer.log
</code></pre>
<p>or:</p>
<pre><code>/var/logs/proxy.log
</code></pre>
<p>Can I use <code>syslog</code> config to do that? What would be an example of that config?</p>
<p>I have already tried <code>journald</code> configuration <code>forward to syslogs=yes</code>.</p>

karlos
<ol>
<li><p>Just first what comes to my mind - create sidecar container that will gather all the logs in 1 place.</p>
</li>
<li><p><a href="https://sematext.com/guides/kubernetes-logging/#kubernetes-logging-architecture-types-of-kubernetes-logs" rel="nofollow noreferrer">The Complete Guide to Kubernetes Logging</a>.</p>
</li>
</ol>
<p>That's a pretty wide question that should be divided on few parts. Kubernets stores different types of logs in different places.</p>
<ul>
<li><p><strong>Kubernetes Container Logs</strong> (out of this question, but simply <code>kubectl logs &lt;podname&gt;</code> + <code>-n</code>  for namespace, if its not default + <code>-c</code> for specifying container inside the pod)</p>
</li>
<li><p><strong>Kubernetes Node Logs</strong></p>
</li>
<li><p><strong>Kubernetes Cluster Logs</strong></p>
</li>
</ul>
<hr />
<p><strong>Kubernetes Node Logs</strong></p>
<blockquote>
<p>Depending on your operating system and services, there are various
node-level logs you can collect, such as kernel logs or systemd logs.</p>
<p>On nodes with systemd both the kubelet and container runtime write to
journald. If systemd is not present, they write to .log files in the
/var/log directory.</p>
<p>You can access systemd logs with the journalctl command.</p>
</blockquote>
<p><a href="https://sematext.com/blog/journald-logging-tutorial/" rel="nofollow noreferrer">Tutorial: Logging with journald</a> have a huge explanation  how can you configure journalctl to gather logs. With agrregation logs tools like ELK and without them. <a href="https://sematext.com/blog/journald-logging-tutorial/#toc-journald-log-filtering-7" rel="nofollow noreferrer">journald log filtering</a> can simplify your life.</p>
<p>There are two ways of centralizing journal entries via syslog:</p>
<ol>
<li><strong>syslog daemon acts as a journald client</strong> (like journalctl or Logstash or Journalbeat)</li>
<li><strong>journald forwards messages to syslog</strong> (via socket)</li>
</ol>
<p>Option 1) is slower – reading from the journal is slower than reading from the socket – but captures all the fields from the journal.</p>
<p>Option 2) is safer (e.g. no issues with journal corruption), but the journal will only forward traditional syslog fields (like severity, hostname, message..)</p>
<p>Talking about <code>ForwardToSyslog=yes</code> in <code>/etc/systemd/journald.conf</code> --&gt; it will write messages, in syslog format, to <code>/run/systemd/journal/syslog</code>. You can pass processing then this file to <a href="https://www.rsyslog.com/" rel="nofollow noreferrer">rsyslog</a> for example. Either you can manually process logs or  move them to desired place..</p>
<hr />
<p><strong>Kubernetes Cluster Logs</strong></p>
<p>By default, system components outside a container write files to journald, while components running in containers write to /var/log directory. However, there is the option to configure the container engine to stream logs to a preferred location.</p>
<p><strong>Kubernetes doesn’t provide a native solution for logging at cluster level</strong>. However, there are other approaches available to you:</p>
<ul>
<li><p>Use a node-level logging agent that runs on every node</p>
</li>
<li><p>Add a sidecar container for logging within the application pod</p>
</li>
<li><p>Expose logs directly from the application.</p>
</li>
</ul>
<hr />
<p>P.S. I have <strong>NOT</strong> tried below approach, but it looks promising - check it and maybe it will help you in your not easiest task.</p>
<blockquote>
<p>The easiest way of setting up a node-level logging agent is to
configure a DaemonSet to run the agent on each node</p>
</blockquote>
<pre><code>helm install --name st-agent \
  --set infraToken=xxxx-xxxx \
  --set containerToken=xxxx-xxxx \
  --set logsToken=xxxx-xxxx \
  --set region=US \
  stable/sematext-agent
</code></pre>
<blockquote>
<p>This setup will, by default, send all cluster and container logs to a
central location for easy management and troubleshooting. With a tiny
bit of added configuration, you can configure it to collect node-level
logs and audit logs as well.</p>
</blockquote>
