<p>I'm pretty much new to Kubernetes and don't have hands-on experience on it. </p>

<p>My team is facing issue regarding the log format pushed by <a href="/questions/tagged/kubernetes" class="post-tag" title="show questions tagged &#39;kubernetes&#39;" rel="tag"><img src="https://i.stack.imgur.com/aIElQ.png" height="16" width="18" alt="" class="sponsor-tag-img">kubernetes</a> to <a href="/questions/tagged/splunk" class="post-tag" title="show questions tagged &#39;splunk&#39;" rel="tag">splunk</a>. </p>

<h3>Application is pushing log to stdout in this format</h3>

<pre><code>{"logname" : "app-log", "level" : "INFO"}
</code></pre>

<h3>Splunk eventually get this format (splunkforwarder is used)</h3>

<pre><code>{
  "log" : "{\"logname\": \"app-log\", \"level\": \"INFO \"}",
  "stream" : "stdout",
  "time" : "2018-06-01T23:33:26.556356926Z" 
 }
</code></pre>

<p>This format kind of make things harder in Splunk to query based on properties.</p>

<p>Is there any options in <code>Kubernetes</code> to forward <code>raw logs</code> from app rather than grouping into another json ?</p>

<p>I came across <a href="https://answers.splunk.com/answers/592640/kubernetesdocker-json-logs.html" rel="nofollow noreferrer">this</a> post in Splunk, but the configuration is done on Splunk side </p>

<p>Please let me know if we have any option from <code>Kubernetes</code> side to send raw logs from application</p>

<p>Kubernetes architecture provides three ways to gather logs:</p>

<p><strong>1. Use a node-level logging agent that runs on every node.</strong></p>

<p>You can implement cluster-level logging by including a node-level logging agent on each node. The logging agent is a dedicated tool that exposes logs or pushes logs to a backend. Commonly, the logging agent is a container that has access to a directory with log files from all of the application containers on that node.</p>

<p>The logs format depends on Docker settings. You need to set up <code>log-driver</code> parameter in <code>/etc/docker/daemon.json</code> on every node.</p>

<p>For example,</p>

<pre><code>{
  "log-driver": "syslog"
}
</code></pre>

<p>or</p>

<pre><code>{
  "log-driver": "json-file"
}
</code></pre>

<ul>
<li>none - no logs are available for the container and docker logs does not
return any output. </li>
<li>json-file - the logs are formatted as JSON. The
default logging driver for Docker. </li>
<li>syslog - writes logging messages to
the syslog facility.</li>
</ul>

<p>For more options, check the <a href="https://docs.docker.com/config/containers/logging/configure/#supported-logging-drivers" rel="nofollow noreferrer">link</a></p>

<p><strong>2. Include a dedicated sidecar container for logging in an application pod.</strong></p>

<p>You can use a sidecar container in one of the following ways:</p>

<ul>
<li>The sidecar container streams application logs to its own stdout. </li>
<li>The sidecar container runs a logging agent, which is configured to pick up logs from an application container.</li>
</ul>

<p>By having your sidecar containers stream to their own stdout and stderr streams, you can take advantage of the kubelet and the logging agent that already run on each node. The sidecar containers read logs from a file, a socket, or the journald. Each individual sidecar container prints log to its own stdout or stderr stream.</p>

<p><strong>3. Push logs directly to a backend from within an application.</strong></p>

<p>You can implement cluster-level logging by exposing or pushing logs directly from every application.</p>

<p>For more information, you can check <a href="https://kubernetes.io/docs/concepts/cluster-administration/logging/#cluster-level-logging-architectures" rel="nofollow noreferrer">official documentation</a> of Kubernetes</p>
