<p>Pods on our k8s cluster are scheduled with <code>Airflow</code>'s <code>KubernetesExecutor</code>, which runs all <code>Task</code>s in a new pod.</p>

<p>I have a such a <code>Task</code> for which the pod instantly (after 1 or 2 seconds) crashes, and for which of course I want to see the logs. </p>

<p>This seems hard. As soon the pod crashes, it gets deleted, along with the ability to retrieve crash logs. I already tried all of: </p>

<ul>
<li><code>kubectl logs -f &lt;pod&gt; -p</code>: cannot be used since these pods are named uniquely
(courtesy of <code>KubernetesExecutor</code>).   </li>
<li><code>kubectl logs -l label_name=label_value</code>: I
struggle to apply the labels to the pod (if this is a known/used way of working, I'm happy to try further)</li>
<li>An shared <code>nfs</code> is mounted on all pods on a fixed log directory. The failing pod however, does not log to this folder.</li>
<li>When I am really quick I run <code>kubectl logs -f -l dag_id=sample_dag --all-containers (</code>dag_id<code>label is added by</code>Airflow<code>)
</code> between running and crashing and see <code>Error from server (BadRequest): container "base" in pod "my_pod" is waiting to start: ContainerCreating</code>. This might give me some clue but:

<ul>
<li>these are only but the last log lines</li>
<li>this is really backwards</li>
</ul></li>
</ul>

<p>I'm basically looking for the canonical way of retrieving logs from transient pods</p>

Raf
<p>You need to enable remote logging. Code sample below is for using S3. In <code>airflow.cfg</code> set the following:</p>

<pre><code>remote_logging = True
remote_log_conn_id = my_s3_conn
remote_base_log_folder = s3://airflow/logs
</code></pre>

<p>The <code>my_s3_conn</code> can be set in airflow>Admin>Connections. In the <code>Conn Type</code> dropdown, select <code>S3</code>.</p>
