<p>I have started using KubernetesExecutor and I have set up a PV/PVC with an AWS EFS to store logs for my dags. I am also using s3 remote logging.</p>
<p>All the logging is working perfectly fine after a dag completes. However, I want to be able to see the logs of my jobs as they are running for long running ones.</p>
<p>When I exec into my scheduler pod, while an executor pod is running, I am able to see the <code>.log</code> file of the currently running job because of the shared EFS. However, when I <code>cat</code> the log file, I do not see the logs as long as the executor is still running. Once the executor finishes however, I can see the full logs both when I <code>cat</code> the file and in the airflow UI.</p>
<p>Weirdly, on the other hand, when I exec into the executor pod as it is running, and I <code>cat</code> the exact same log file in the shared EFS, I am able to see the correct logs up until that point in the job, and when I immediately <code>cat</code> from the scheduler or check the UI, I can also see the logs up until that point.</p>
<p>So it seems that when I <code>cat</code> from within the executor pod, it is causing the logs to be flushed in some way, so that it is available everywhere. Why are the logs not flushing regularly?</p>
<p>Here are the config variables I am setting, note these env variables get set in my webserver/scheduler and executor pods:</p>
<pre><code># ----------------------
# For Main Airflow Pod (Webserver &amp; Scheduler)
# ----------------------
export PYTHONPATH=$HOME
export AIRFLOW_HOME=$HOME
export PYTHONUNBUFFERED=1

# Core configs
export AIRFLOW__CORE__LOAD_EXAMPLES=False
export AIRFLOW__CORE__SQL_ALCHEMY_CONN=${AIRFLOW__CORE__SQL_ALCHEMY_CONN:-postgresql://$DB_USER:$DB_PASSWORD@$DB_HOST:5432/$DB_NAME}
export AIRFLOW__CORE__FERNET_KEY=$FERNET_KEY
export AIRFLOW__CORE__DAGS_FOLDER=$AIRFLOW_HOME/git/dags/$PROVIDER-$ENV/

# Logging configs
export AIRFLOW__LOGGING__BASE_LOG_FOLDER=$AIRFLOW_HOME/logs/
export AIRFLOW__LOGGING__REMOTE_LOGGING=True
export AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID=aws_default
export AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER=s3://path-to-bucket/airflow_logs
export AIRFLOW__LOGGING__TASK_LOG_READER=s3.task
export AIRFLOW__LOGGING__LOGGING_CONFIG_CLASS=config.logging_config.LOGGING_CONFIG

# Webserver configs
export AIRFLOW__WEBSERVER__COOKIE_SAMESITE=None
</code></pre>
<p>My logging config looks like the one in the question <a href="https://stackoverflow.com/questions/55526759/airflow-1-10-2-not-writing-logs-to-s3">here</a></p>
<p>I thought this could be a python buffering issue so added <code>PYTHONUNBUFFERED=1</code>, but that didn't help. This is happening whether I use the <code>PythonOperator</code> or <code>BashOperator</code></p>
<p>Is it the case that K8sExecutors logs just won't be available during their runtime? Only after? Or is there some configuration I must be missing?</p>

<p>I had the same issue and those are things that helped me - worth checking them on your end</p>
<ul>
<li><code>PYTHONUNBUFFERED=1</code> is not enough, but necessary to view logs in realtime. Please keep it</li>
<li>have EFS mounted in web, scheduler, and pod_template (executor).</li>
<li>Your experience with log file being complete after task having finished, makes me wonder if PVC you use for logs, has ReadWriteMany accessMode</li>
<li>Are the paths you cat in different pods, identical? Do they include full task format, eg <code>efs/logs/dag_that_executes_via_KubernetesPodOperator/task1/2021-09-21T19\:00\:21.894859+00\:00/1.log</code> ? Asking because, before i had EFS hooked up in every place (scheduler, web, pod_template), i could only access executor logs that do not include task name and task time</li>
<li>have EFS logs folder belong to airflow (for me uid 50000 because may have to prepare this from different place), group root, mode 755</li>
<li>do not have AIRFLOW__LOGGING__LOGGING_CONFIG_CLASS set up. Try to get things running as vanilla as possible, before introducing custom logging config</li>
</ul>
<p>If you have remote logging set up, i understand that after task completes, the first line in the UI is going to say <code>Reading remote log from</code>, but what does the first line say for you when the task is running? <code>reading remote</code> or mentioning usage of local log file ?</p>
<ul>
<li>If it says about remote, this would mean that you don't have EFS hooked up in every place.</li>
<li>If it says about local, i would check your EFS settings (readwritemany) and directory ownership and mode</li>
</ul>
