<p>I read in kubernetes docs somewhere that kubernetes reads application logs from stdout and stderror in pods. 
I created a new application and configured it to send logs to a remote splunk hec endpoint (using splunk-logback jars) and at the same time to console. 
So by default, the console logs in logback should go to System.out, which should then be visible using kubectl logs .
But it's not happening in my application.</p>

<p>my logback file:</p>

<pre><code>&lt;?xml version="1.0" encoding="UTF-8"?&gt;

&lt;configuration&gt;

    &lt;Appender name="SPLUNK" class="com.splunk.logging.HttpEventCollectorLogbackAppender"&gt;
        &lt;url&gt;${splunk_hec_url}&lt;/url&gt;
        &lt;token&gt;${splunk_hec_token}&lt;/token&gt;
        &lt;index&gt;${splunk_app_token}&lt;/index&gt;
        &lt;disableCertificateValidation&gt;true&lt;/disableCertificateValidation&gt;
        &lt;batch_size_bytes&gt;1000000&lt;/batch_size_bytes&gt;
        &lt;batch_size_count&gt;${batch_size_count}&lt;/batch_size_count&gt;
        &lt;send_mode&gt;sequential&lt;/send_mode&gt;

        &lt;layout class="ch.qos.logback.classic.PatternLayout"&gt;
            &lt;pattern&gt;%msg&lt;/pattern&gt;
        &lt;/layout&gt;
    &lt;/Appender&gt;

    &lt;Appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender"&gt;
        &lt;encoder&gt;
            &lt;pattern&gt;%msg&lt;/pattern&gt;
        &lt;/encoder&gt;
    &lt;/Appender&gt;

    &lt;Appender name="ASYNC" class="ch.qos.logback.classic.AsyncAppender"&gt;
        &lt;appender-ref ref="STDOUT" /&gt;
    &lt;/Appender&gt;

    &lt;root level="INFO"&gt;
        &lt;appender-ref ref="SPLUNK"/&gt;
        &lt;appender-ref ref="ASYNC"/&gt;
    &lt;/root&gt;

&lt;/configuration&gt;
</code></pre>

<p>I am able to see the logs in splunk and If I login to the container from backend and start my java application, then also I can see the logs on the terminal that time. But if I let the container start by default on it's own, then the logs are only going to splunk and I can't view them using <code>kubectl logs &lt;POD_NAME&gt;</code></p>

<p>The kubernetes yml file for my logger app:</p>

<pre><code>apiVersion: v1
kind: Pod
metadata:
    name: logging-pod
    labels:
       app: logging-pod
spec:
  containers:
     - name: logging-container
       image: logger-splunk:latest
       command: ["java", "-jar", "logger-splunk-1.0-SNAPSHOT.jar"]
       resources:
          requests:
             cpu: 1
             memory: 1Gi
          limits:
             cpu: 1
             memory: 1Gi
</code></pre>

<p>According to the Kubenetes <a href="https://kubernetes.io/docs/concepts/cluster-administration/logging/" rel="nofollow noreferrer">documentation</a>, all output (that a containerized application writes to <code>stdout</code> and <code>stderr</code>) is redirected to a JSON file by default. You can access it by using <code>kubectl logs</code>. </p>

<p>Let's test this feature by creating a simple pod that outputs numbers in stdout:</p>

<pre><code>kubectl create -f https://k8s.io/docs/tasks/debug-application-cluster/counter-pod.yaml
</code></pre>

<p><em>counter-pod.yaml:</em></p>

<pre><code>apiVersion: v1
kind: Pod
metadata:
  name: counter
spec:
  containers:
  - name: count
    image: busybox
    args: [/bin/sh, -c,
            'i=0; while true; do echo "$i: $(date)"; i=$((i+1)); sleep 1; done']
</code></pre>

<p>where:<br>
 <code>counter</code> - name of the pod<br>
 <code>count</code>   - name of the container inside "counter" pod</p>

<p>You can access the content of that file by running:  </p>

<pre><code>$ kubectl logs counter
</code></pre>

<p>You can access a log file of previously crashed container in a pod by the following command:  </p>

<pre><code>$ kubectl logs --previous
</code></pre>

<p>In case of multiple containers in the pod, you should add the name of the container as follows:</p>

<pre><code>$ kubectl logs counter -c count
</code></pre>

<p>When the pod is removed from the cluster, all its logs (current and previous) are also removed.</p>

<p>Ensure you configure stdout in application correctly, and the output to stdout in your application is not silently skipped by any reason.</p>
