title: Kubernetes Executor does not spawn Kubernetes Pod Operator
body: <p>can't figure out where to look. I am running Airflow on GKE. It was running fine but recently started failing. Can't understand where to look. Basically, DAG starts, and then tasks fail, but they were running okay a week ago.</p>
  <p>It seems like something changed in a cluster, but based on logs, can't figure out what.</p>
  <p>My <code>KubernetesExecutor</code> stopped spawning <code>KubernetesPodOperators</code> and there are no logs or errors.
  If I run directly (<code>kubectl apply -f</code>) template I use for Operator, it runs successfully.</p>
  <p><strong>Airflow</strong> 2.1.2</p>
  <p><strong>Kubectl</strong></p>
  <pre><code>Client Version: version.Info{Major:&quot;1&quot;, Minor:&quot;20&quot;, GitVersion:&quot;v1.20.7&quot;, GitCommit:&quot;132a687512d7fb058d0f5890f07d4121b3f0a2e2&quot;, GitTreeState:&quot;clean&quot;, BuildDate:&quot;2021-05-12T12:40:09Z&quot;, GoVersion:&quot;go1.15.12&quot;, Compiler:&quot;gc&quot;, Platform:&quot;darwin/amd64&quot;}
  Server Version: version.Info{Major:&quot;1&quot;, Minor:&quot;20+&quot;, GitVersion:&quot;v1.20.8-gke.900&quot;, GitCommit:&quot;28ab8501be88ea42e897ca8514d7cd0b436253d9&quot;, GitTreeState:&quot;clean&quot;, BuildDate:&quot;2021-06-30T09:23:36Z&quot;, GoVersion:&quot;go1.15.13b5&quot;, Compiler:&quot;gc&quot;, Platform:&quot;linux/amd64&quot;}
  </code></pre>
  <p><strong>Executor Template</strong></p>
  <pre class="lang-yaml prettyprint-override"><code>apiVersion: v1
  kind: Pod
  metadata:
    ...
  spec:
    restartPolicy: Never
    serviceAccountName: airflow # this account have rights to create pods
    automountServiceAccountToken: true
    volumes:
        - name: dags
          emptyDir: {}
        - name: logs
          emptyDir: {}
        - configMap:
            name: airflow-git-sync-configmap
          name: airflow-git-sync-configmap
    initContainers:
      - name: git-sync-clone
        securityContext:
          runAsUser: 65533
          runAsGroup: 65533
        image: k8s.gcr.io/git-sync/git-sync:v3.3.1
        imagePullPolicy: Always
        volumeMounts:
          - mountPath: /tmp/git
            name: dags
        resources:
           ...
        args: [&quot;--one-time&quot;]
        envFrom:
        - configMapRef:
            name: airflow-git-sync-configmap
        - secretRef:
            name: airflow-git-sync-secret
    containers:
      - name: base
        image: &lt;artifactory_url&gt;/airflow:latest
        volumeMounts:
          - name: dags
            mountPath: /opt/airflow/dags
          - name: logs
            mountPath: /opt/airflow/logs
        imagePullPolicy: Always
  </code></pre>
  <p><strong>Pod template</strong></p>
  <pre class="lang-yaml prettyprint-override"><code>apiVersion: v1
  kind: Pod
  metadata:
     ....
  spec:
    serviceAccountName: airflow
    automountServiceAccountToken: true
    volumes:
      - name: sql
        emptyDir: {}
    initContainers:
      - name: git-sync
        image: k8s.gcr.io/git-sync/git-sync:v3.3.1
        imagePullPolicy: Always
        args: [&quot;--one-time&quot;]
        volumeMounts:
          - name: sql
            mountPath: /tmp/git/
        resources:
          requests:
            memory: 300Mi
            cpu: 500m
          limits:
            memory: 600Mi
            cpu: 1000m
        envFrom:
        - configMapRef:
            name: git-sync-configmap
        - secretRef:
            name: airflow-git-sync-secret
    containers:
      - name: base
        imagePullPolicy: Always
        image: &lt;artifactory_url&gt;/clickhouse-client-gcloud:20.6.4.44
        volumeMounts:
          - name: sql
            mountPath: /opt/sql
        resources:
           ....
        env:
          - name: GS_SERVICE_ACCOUNT
            valueFrom:
              secretKeyRef:
                name: gs-service-account
                key: service_account.json
          - name: DB_CREDENTIALS
            valueFrom:
              secretKeyRef:
                name: estimation-db-secret
                key: db_cred.json
  </code></pre>
  <p><strong>DAG code</strong></p>
  <pre class="lang-py prettyprint-override"><code>from textwrap import dedent
  
  from airflow import DAG
  from airflow.utils.dates import days_ago
  from airflow.contrib.operators.kubernetes_pod_operator import KubernetesPodOperator
  
  TEMPLATE_PATH = &quot;/opt/airflow/dags/airflow-dags.git/pod_templates&quot;
  
  args = {
      ...
  }
  
  
  def create_pipeline(dag_: DAG):
  
      task_startup_client = KubernetesPodOperator(
          name=&quot;clickhouse-client&quot;,
          task_id=&quot;clickhouse-client&quot;,
          labels={&quot;application&quot;: &quot;clickhouse-client-gsutil&quot;},
          pod_template_file=f&quot;{TEMPLATE_PATH}/template.yaml&quot;,
          cmds=[&quot;sleep&quot;, &quot;60000&quot;],
          reattach_on_restart=True,
          is_delete_operator_pod=False,
          get_logs=True,
          log_events_on_failure=True,
          dag=dag_,
      )
    
      task_startup_client
  
  
  with DAG(
      dag_id=&quot;MANUAL-GKE-clickhouse-client&quot;,
      default_args=args,
      schedule_interval=None,
      max_active_runs=1,
      start_date=days_ago(2),
      tags=[&quot;utility&quot;],
  ) as dag:
  
      create_pipeline(dag)
  </code></pre>
  <p>I ran Airflow with <code>DEBUG</code> logging and there is nothing, successful completion:</p>
  <p><strong>Scheduler log</strong></p>
  <pre><code>...
  Event: manualgkeclickhouseclientaticlickhouseclient.9959fa1fd13a4b6fbdaf40549a09d2f9 Succeeded
  ...
  </code></pre>
  <p>*<strong>Executor logs</strong></p>
  <pre><code>[2021-08-15 18:40:27,045] {settings.py:208} DEBUG - Setting up DB connection pool (PID 1)
  [2021-08-15 18:40:27,046] {settings.py:276} DEBUG - settings.prepare_engine_args(): Using pool settings. pool_size=5, max_overflow=10, pool_recycle=1800, pid=1
  [2021-08-15 18:40:27,095] {cli_action_loggers.py:40} DEBUG - Adding &lt;function default_action_log at 0x7f0556c5e280&gt; to pre execution callback
  [2021-08-15 18:40:28,070] {cli_action_loggers.py:66} DEBUG - Calling callbacks: [&lt;function default_action_log at 0x7f0556c5e280&gt;]
  [2021-08-15 18:40:28,106] {settings.py:208} DEBUG - Setting up DB connection pool (PID 1)
  [2021-08-15 18:40:28,107] {settings.py:244} DEBUG - settings.prepare_engine_args(): Using NullPool
  [2021-08-15 18:40:28,109] {dagbag.py:496} INFO - Filling up the DagBag from /opt/airflow/dags/ati-airflow-dags.git/dag_clickhouse-client.py
  [2021-08-15 18:40:28,110] {dagbag.py:311} DEBUG - Importing /opt/airflow/dags/ati-airflow-dags.git/dag_clickhouse-client.py
  /usr/local/lib/python3.9/site-packages/airflow/providers/cncf/kubernetes/backcompat/backwards_compat_converters.py:26 DeprecationWarning: This module is deprecated. Please use `kubernetes.client.models.V1Volume`.
  /usr/local/lib/python3.9/site-packages/airflow/providers/cncf/kubernetes/backcompat/backwards_compat_converters.py:27 DeprecationWarning: This module is deprecated. Please use `kubernetes.client.models.V1VolumeMount`.
  [2021-08-15 18:40:28,135] {dagbag.py:461} DEBUG - Loaded DAG &lt;DAG: MANUAL-GKE-clickhouse-client&gt;
  [2021-08-15 18:40:28,176] {plugins_manager.py:281} DEBUG - Loading plugins
  [2021-08-15 18:40:28,176] {plugins_manager.py:225} DEBUG - Loading plugins from directory: /opt/airflow/plugins
  [2021-08-15 18:40:28,177] {plugins_manager.py:205} DEBUG - Loading plugins from entrypoints
  [2021-08-15 18:40:28,238] {plugins_manager.py:418} DEBUG - Integrate DAG plugins
  Running &lt;TaskInstance: MANUAL-GKE-clickhouse-client.clickhouse-client 2021-08-15T18:39:38.150950+00:00 [queued]&gt; on host manualgkeclickhouseclientclickhouseclient.9959fa1fd13a4b6fbd
  [2021-08-15 18:40:28,670] {cli_action_loggers.py:84} DEBUG - Calling callbacks: []
  [2021-08-15 18:40:28,670] {settings.py:302} DEBUG - Disposing DB connection pool (PID 1)
  </code></pre>