<p>I'm running Apache Airflow on a Kubernetes cluster and I'm facing an issue with retrieving logs from deleted pods. I can easily get logs from currently running or recently terminated pods using kubectl logs, but I'm unable to get logs from older, deleted pods.</p>
<h3>My versions</h3>
<pre><code>awswrangler==2.19.0
apache-airflow-providers-cncf-kubernetes==4.3.0
apache-airflow-providers-amazon==5.0.0
boto3==1.24.56
gnupg==2.3.1
PyYAML==6.0
</code></pre>
<p>Here's what I've tried:</p>
<pre><code># This works fine and gives me the actual log
kubectl logs my-current-pod-239847283947

# This doesn't work for an old, deleted pod
kubectl logs my-old-pod-928374928374
The second command returns the following error:
</code></pre>
<h3>The error</h3>
<pre><code>Error from server (NotFound): pods &quot;my-old-pod&quot; not found
</code></pre>
<p>I understand that the pod is deleted, but is there a way to retrieve its logs or at least configure Airflow or Kubernetes to save these logs for future reference?</p>
<p>NOTE: I'm using AWS for storage</p>
<p>Any help would be greatly appreciated!</p>

The Dan
<p>You might be able to get the logs by following <a href="https://stackoverflow.com/questions/57007134/how-to-see-logs-of-terminated-pods">How to see logs of terminated pods</a></p>
<p>However it's much easier to handle this by simply keep the pod you need.</p>
<p>Airflow offers the option to delete the pod when the task is finished you simply need to disable the deletion.</p>
<p>For <code>apache-airflow-providers-cncf-kubernetes&gt;=7.2.0</code>:</p>
<pre><code>KubernetesPodOperator(..., on_finish_action='keep_pod')
</code></pre>
<p>NOTE: If you don't mind about logs of successful tasks you can also set to <code>on_finish_action='delete_succeeded_pod'</code> which will delete only successful pods thus leaving the errored ones for further investigation. This offers much more flexibility than older versions of the provider (See <a href="https://github.com/apache/airflow/pull/30718" rel="nofollow noreferrer">PR</a>).</p>
<p>For <code>apache-airflow-providers-cncf-kubernetes&lt;7.2.0</code>:</p>
<pre><code>KubernetesPodOperator(..., is_delete_operator_pod=False)
</code></pre>
<p>If you no longer need the pods you can delete them with kubectl. You can set also script (to be executed with Airflow) that will clean older pods (for example delete all pods older than X days).</p>
