<p>I have a very simple setup.
My .yaml-Configuration file for Kubernetes:</p>
<pre><code>apiVersion: batch/v1
kind: Job
metadata:
  name: stdout-test
spec:
  template:
    spec:
      priorityClassName: research-high
      containers:
        - name: container-stdout-test
          image: &lt;here comes my secret image repo&gt;
          imagePullPolicy: &quot;IfNotPresent&quot;
          resources:
            limits:
              nvidia.com/gpu: &quot;1&quot;
              cpu: &quot;1&quot;
              memory: &quot;8Gi&quot;
            requests:
              nvidia.com/gpu: &quot;1&quot;
              cpu: &quot;1&quot;
              memory: &quot;4Gi&quot;
          command: [&quot;python3&quot;, &quot;/workspace/main.py&quot;]
          volumeMounts:
            - mountPath: /workspace 
              name: localdir 
      imagePullSecrets:
        - name: lsx-registry
      restartPolicy: &quot;Never&quot;
      volumes:
        - name: localdir
          cephfs:
            monitors:
              - &lt;here come my secret monitors&gt;
            user: &lt;namespace&gt;
            path: &quot;/home/stud/nothelfer/stdout-test&quot; 
            secretRef: 
              name: &lt;my secret&gt;
</code></pre>
<p>And my simple python program (<code>main.py</code>):</p>
<pre><code>import time

for i in range(0, 1000):
    print(i)
    time.sleep(1.0)
</code></pre>
<p>Starting and running the job in kubernetes works fine. I get this output from <code>kubectl get pods</code>:</p>
<pre><code>NAME                READY   STATUS    RESTARTS   AGE
stdout-test-hl6qs   1/1     Running   0          5s
</code></pre>
<p>But i dont get the expected output 0, 1, 2, ... from stdout by using <code>kubectl logs -f stdout-test-hl6qs</code>. Instead i just get a blank screen. There is just no output printed to the console. I was expecting to get the output of my simple python program. I have tried all possible command line arguments for <code>kubectl logs</code> but none of them make me see the output of my Python program in the command line. The image is set up correctly, the Python program runs fine in the container, I have already checked all that. Can anyone help me here?</p>
<p>My Dockerfile:</p>
<pre><code>FROM pytorch/pytorch
RUN conda install matplotlib pandas numpy
RUN conda update --all

ENV PYTHONUNBUFFERED 1

RUN python -c &quot;import torch, pandas, matplotlib&quot;
</code></pre>

<p>After quite a few tries, I figured out how to see the stdout output. In the Kubernetes configuration file, I had to replace line <code>command: [&quot;python3&quot;, &quot;/workspace/main.py&quot;]</code> with <code>command: [&quot;python3&quot;, &quot;-u&quot;, &quot;/workspace/main.py&quot;]</code>, which configures the output from Python to unbuffered. For reasons that aren't entirely clear, it didn't work when I only set variable <code>ENV PYTHONUNBUFFERED=1</code> or <code>ENV PYTHONUNBUFFERED 1</code> in the Dockerfile. I can now see the output of the program whether I specified <code>ENV PYTHONUNBUFFERED=1</code> or <code>ENV PYTHONUNBUFFERED 1</code> in the Dockerfile or not.</p>
<p><a href="https://docs.python.org/3/using/cmdline.html#envvar-PYTHONUNBUFFERED" rel="nofollow noreferrer">PYTHONUNBUFFERED</a></p>
