<p>I am running airflow via MWAA on aws and the worker nodes are running k8s. The pods are getting scheduled just fine but I am trying to use pod_template_file with KubernetesPodOperator, it's giving me a bunch of uncertain behavior.</p>
<p>My template file stored in S3</p>
<pre><code>apiVersion: v1
kind: Pod
metadata:
  name: app1
  namespace: app1
spec:
  containers:
  - name: base
    image: &quot;alpine:latest&quot;
    command: [&quot;/bin/sh&quot;]
    args: [&quot;-c&quot;, &quot;while true; do echo hi&gt;&gt; /data/app.log; sleep 5; done&quot;]
    volumeMounts:
    - name: persistent-storage
      mountPath: /data
  volumes:
  - name: persistent-storage
    persistentVolumeClaim:
      claimName: claim1
</code></pre>
<p>My DAG file</p>
<pre><code>t_1 = KubernetesPodOperator(
    task_id=job_name_1,
    namespace=&quot;app&quot;,
    name=job_name_1,
    get_logs=True,
    is_delete_operator_pod=False,
    in_cluster=False,
    config_file=kube_config_path,
    startup_timeout_seconds=240,
    cluster_context='test',
    pod_template_file=&quot;/opt/airflow/pod_template_file_example-1.yaml&quot;,
    dag=dag)
</code></pre>
<p>When I go with this, I get an error that the pod spec is invalid as it's missing image field. This is surprising as <code>image</code> is present in pod-template.</p>
<p>I also tried below, it works, but it totally ignores the pod-template file and spins up a alpine container and exits. So looks like it's totally ignoring the pod_template_file param.</p>
<pre><code>full_pod_spec = k8s.V1Pod(
    metadata=metadata_2,
    spec=k8s.V1PodSpec(containers=[
        k8s.V1Container(
            name=&quot;base&quot;,
            image=&quot;alpine:latest&quot;,
        )
    ], ))

t_1 = KubernetesPodOperator(
    task_id=job_name_1,
    namespace=&quot;mlops&quot;,
    name=job_name_1,
    get_logs=True,
    is_delete_operator_pod=False,
    in_cluster=False,
    config_file=kube_config_path,
    startup_timeout_seconds=240,
    cluster_context='aws',
    full_pod_spec=full_pod_spec,
    pod_template_file=&quot;/opt/airflow/pod_template_file_example-1.yaml&quot;,
    dag=dag)
</code></pre>
<p>What is the correct way to reference a pod_template_file in KubernetesPodOperator in airflow?</p>
<p>References : <a href="https://imsharadmishra.medium.com/different-ways-to-use-kubernetes-pod-operator-in-airflow-74e655221386" rel="nofollow noreferrer">medium</a></p>

<p>Never used <code>MWAA</code> on aws before. But your problem seems to be:</p>
<p><strong>Is your template file stored in S3 available to the DAG script file through path <code>&quot;/opt/airflow/pod_template_file_example-1.yaml&quot;</code>?</strong></p>
<p>Check out the docs for <a href="https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/stable/operators.html#difference-between-kubernetespodoperator-and-kubernetes-object-spec" rel="nofollow noreferrer">Difference between KubernetesPodOperator and Kubernetes object spec</a>.</p>
<p>So <code>KubernetesPodOperator</code> just creates the equivalent YAML/JSON object spec for the Pod, and it contains 4 types of arguments for this purpose with the following order of precedence:</p>
<ol>
<li>KubernetesPodOperator arguments, like <code>name/namespace/image</code>...;</li>
<li>full pod spec, which is arg: <code>full_pod_spec</code>;</li>
<li>pod template file, which is arg: <code>pod_template_file</code>;</li>
<li>airflow connection;</li>
</ol>
