<p>I'm trying to run a task in an environment built from an image in a private Google Container Registry through the KubernetesPodOperator of the Google Cloud Composer.
The Container Registry and Cloud Composer instances are under the same project.
My code is below.</p>
<pre><code>import datetime
import airflow
from airflow.contrib.operators import kubernetes_pod_operator


YESTERDAY = datetime.datetime.now() - datetime.timedelta(days=1)


# Create Airflow DAG the the pipeline
with airflow.DAG(
        'my_dag',
        schedule_interval=datetime.timedelta(days=1),
        start_date=YESTERDAY) as dag:

    my_task = kubernetes_pod_operator.KubernetesPodOperator(
        task_id='my_task',
        name='my_task',
        cmds=['echo 0'],
        namespace='default',
        image=f'gcr.io/&lt;my_private_repository&gt;/&lt;my_image&gt;:latest')
</code></pre>
<p>The task fails and I get the following error message in the logs in the Airflow UI and in the <code>logs</code> folder in the storage bucket.</p>
<pre><code>[2020-09-21 08:39:12,675] {taskinstance.py:1147} ERROR - Pod Launching failed: Pod returned a failure: failed
Traceback (most recent call last)
  File &quot;/usr/local/lib/airflow/airflow/contrib/operators/kubernetes_pod_operator.py&quot;, line 260, in execut
    'Pod returned a failure: {state}'.format(state=final_state
airflow.exceptions.AirflowException: Pod returned a failure: failed
</code></pre>
<p>This not very informative...
Any idea what I could be doing wrong?
Or anywhere I can find more informative log messages?</p>
<p>Thank you very much!</p>

Théophile Gervet
<p>In general, the way how we start troubleshooting GCP Composer once getting a failure running the DAG is finely explained in the dedicated <a href="https://cloud.google.com/composer/docs/how-to/using/troubleshooting-dags" rel="nofollow noreferrer">chapter</a> of GCP documentation.</p>
<p>Moving to <a href="https://cloud.google.com/composer/docs/how-to/using/using-kubernetes-pod-operator" rel="nofollow noreferrer">KubernetesPodOperator</a> specifically related issues, the certain user investigation might consists of:</p>
<ul>
<li>Verifying the particular <a href="https://cloud.google.com/composer/docs/how-to/using/using-kubernetes-pod-operator#task-status" rel="nofollow noreferrer"><em>task status</em></a> for the corresponded DAG
file;</li>
<li>Inspecting the task inventory <a href="https://cloud.google.com/composer/docs/how-to/using/using-kubernetes-pod-operator#task-logs" rel="nofollow noreferrer"><em>logs and events;</em></a>, logs also can be found in GCP Composer's <a href="https://cloud.google.com/composer/docs/concepts/logs#logs_in" rel="nofollow noreferrer">storage bucket</a>;</li>
<li>With any K8s related <a href="https://cloud.google.com/stackdriver/docs/solutions/gke/using-logs#resource_types" rel="nofollow noreferrer">resource/objects</a> errors it's strongly
required to check Composer's relevant <a href="https://cloud.google.com/stackdriver/docs/solutions/gke/using-logs#finding_your_logs_in_logs_viewer" rel="nofollow noreferrer">GKE cluster</a> log/event
journals.</li>
</ul>
<p>Further analyzing the error context and <code>KubernetesPodOperator.py</code> source code, I assume that this issue might occur due to Pod launching problem on Airflow worker GKE node, ending up with <code>Pod returned a failure: {state}'.format(state=final_state)</code> message once the Pod execution is not successful.</p>
<p>Personally, I prefer to check the image run in prior executing Airflow task in a Kubernetes Pod. Having said this and based on the task command provided, I believe that you can verify the Pod launching process, <a href="https://cloud.google.com/composer/docs/how-to/managing/deploy-webserver#connect-cluster" rel="nofollow noreferrer"><em>connecting to GKE</em></a> cluster and redrafting <code>kubernetes_pod_operator.KubernetesPodOperator</code> definition being adoptable for <code>kubectl </code> command-line executor:</p>
<pre><code>kubectl run test-app --image=eu.gcr.io/&lt;Project_ID&gt;/image --command -- &quot;/bin/sh&quot; &quot;-c&quot; &quot;echo 0&quot;
</code></pre>
<p>This would simplify the process of image validation, hence you'll be able to get closer look at Pod logs or event records as well:</p>
<pre><code>kubectl describe po test-app
</code></pre>
<p>Or</p>
<pre><code>kubectl logs test-app
</code></pre>
