<p>I'm looking to redirect some logs from a command run with <code>kubectl exec</code> to that pod's logs, so that they can be read with <code>kubectl logs &lt;pod-name&gt;</code> (or really, <code>/var/log/containers/&lt;pod-name&gt;.log</code>). I can see the logs I need as output when running the command, and they're stored inside a separate log directory inside the running container.  </p>

<p>Redirecting the output (i.e. <code>&gt;&gt; logfile.log</code>) to the file which I thought was mirroring what is in <code>kubectl logs &lt;pod-name&gt;</code> does not update that container's logs, and neither does redirecting to stdout.</p>

<p>When calling <code>kubectl logs &lt;pod-name&gt;</code>, my understanding is that kubelet gets them from it's internal <code>/var/log/containers/</code> directory.  But what determines which logs are stored there? Is it the same process as the way logs get stored inside any other docker container?</p>

<p>Is there a way to examine/trace the logging process, or determine where these logs are coming from?</p>

oandersonm
<p>Logs from the <code>STDOUT</code> and <code>STDERR</code> of containers in the pod are captured and stored inside files in /var/log/containers. This is what is presented when <code>kubectl log</code> is run.</p>

<p>In order to understand why output from commands run by kubectl exec is not shown when running <code>kubectl log</code>, let's have a look how it all works with an example:</p>

<p>First launch a pod running ubuntu that are sleeping forever:</p>

<pre><code>$&gt; kubectl run test --image=ubuntu --restart=Never -- sleep infinity
</code></pre>

<p>Exec into it</p>

<pre><code>$&gt; kubectl exec -it test bash
</code></pre>

<p>Seen from inside the container it is the <code>STDOUT</code> and <code>STDERR</code> of PID 1 that are being captured. When you do a <code>kubectl exec</code> into the container a new process is created living alongside PID 1:</p>

<pre><code>root@test:/# ps -auxf
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         7  0.0  0.0  18504  3400 pts/0    Ss   20:04   0:00 bash
root        19  0.0  0.0  34396  2908 pts/0    R+   20:07   0:00  \_ ps -auxf
root         1  0.0  0.0   4528   836 ?        Ss   20:03   0:00 sleep infinity
</code></pre>

<p>Redirecting to <code>STDOUT</code> is not working because <code>/dev/stdout</code> is a symlink to the process accessing it (<code>/proc/self/fd/1</code> rather than <code>/proc/1/fd/1</code>).</p>

<pre><code>root@test:/# ls -lrt /dev/stdout
lrwxrwxrwx 1 root root 15 Nov  5 20:03 /dev/stdout -&gt; /proc/self/fd/1
</code></pre>

<p>In order to see the logs from commands run with <code>kubectl exec</code> the logs need to be redirected to the streams that are captured by the kubelet (<code>STDOUT</code> and <code>STDERR</code> of pid 1). This can be done by redirecting output to <code>/proc/1/fd/1</code>.</p>

<pre><code>root@test:/# echo "Hello" &gt; /proc/1/fd/1
</code></pre>

<p>Exiting the interactive shell and checking the logs using <code>kubectl logs</code> should now show the output</p>

<pre><code>$&gt; kubectl logs test
Hello
</code></pre>
