<p>I am attempting to migrate an airflow deployment running in kubernetes from the <code>CeleryExecutor</code> to the <code>KubernetesExecutor</code>. Everything went smoothly in my local development environment (running on minikube), however I need to load a sidecar container in production to run a proxy that allows me to connect to my sql database. After some googling it appears that defining the <a href="https://airflow.apache.org/docs/stable/kubernetes.html#pod-mutation-hook" rel="nofollow noreferrer">pod_mutation_hook</a> function in an <code>airflow_local_settings.py</code> file somewhere on the <code>$PYTHONPATH</code> is how one is supposed to accomplish this.</p>

<p>First I tried defining this in a config map per <a href="https://github.com/astronomer/airflow-chart/blob/master/templates/configmap.yaml#L131-L152" rel="nofollow noreferrer">this</a> example. e.g.</p>

<pre><code>apiVersion: v1
kind: ConfigMap
metadata:
  name: airflow-config
  namespace: dev
data:
  ...

  AIRFLOW__KUBERNETES__LOGS_VOLUME_CLAIM: "airflow-logs"

  AIRFLOW__KUBERNETES__AIRFLOW_LOCAL_SETTINGS_CONFIGMAP: "airflow-config"
  ...

  airflow_local_settings.py: |
    from airflow.contrib.kubernetes.pod import Pod

    def pod_mutation_hook(pod: Pod):
        extra_labels = {
            "test-label": "True",
        }
        pod.labels.update(extra_labels)
</code></pre>

<p>I specified this configmap in the <code>airflow.cfg</code> file, and it gets picked up and mounted fine, all the other env variables work correctly, but <code>pod_mutation_hook</code> does not appear to run as no labels are added to the resulting pod launched by the kubernetes executor (note that the logs volume claim is also specified here, and works correctly). </p>

<p>Next I tried to define the <code>airflow_local_settings.py</code> file in the image that airflow is launching for the job under <code>$AIRFLOW_HOME/configs/airflow_local_settings.py</code> as suggested in a comment <a href="https://stackoverflow.com/questions/60443124/how-to-set-request-cpu-globally-for-airflow-worker-pods-using-the-kubernetes-exe">here</a>. I also removed the relevant sections from the <code>airflow-config</code> configmap above. This also appeared to have no effect on the resulting pod created for the job, as it also lacked the specified labels.</p>

<p>So, I am unsure how to proceed at this point, because I don't understand how I am supposed to specify the <code>airflow_local_settings.py</code> file and the <code>pod_mutation_hook</code> function such that they actually mutate the pod before running. Any help would be greatly appreciated. Thank you.</p>

<p><strong>Summary</strong>:</p>
<p>You should put your <code>airflow_local_settings.py</code> file on <code>PYTHONPATH</code> for the Scheduler at the very least if you want the sidecars on all pods launched by the KubernetesExecutor or KubernetesPodOperator (with a different Executor) as PODs for both of them are launched by the Scheduler.</p>
<p>However, if you also want sidecars on the PODs launched by <code>KubernetesPodOperator</code> when using <code>KubernetesExecutor</code> you will need to set <code>airflow_local_settings_configmap</code> in <code>airflow.cfg</code> (like done at <a href="https://github.com/astronomer/airflow-chart/blob/f3dddeffe43c92d594cfcfe9c5b001680f45a986/templates/configmap.yaml#L72" rel="nofollow noreferrer">https://github.com/astronomer/airflow-chart/blob/f3dddeffe43c92d594cfcfe9c5b001680f45a986/templates/configmap.yaml#L72</a>) as when you use KubernetePodOperator with KubernetesExecutor, the task pods (with KubernetesPodOperator) will be launched by the worker POD.</p>
<p>Notice how we also pass the same configmap to Scheduler deployment (<a href="https://github.com/astronomer/airflow-chart/blob/f3dddeffe43c92d594cfcfe9c5b001680f45a986/templates/scheduler/scheduler-deployment.yaml#L125-L135" rel="nofollow noreferrer">https://github.com/astronomer/airflow-chart/blob/f3dddeffe43c92d594cfcfe9c5b001680f45a986/templates/scheduler/scheduler-deployment.yaml#L125-L135</a>) and to <code>airflow.cfg</code> itself too since we want all the PODs to mutate via the pod_mutation_hook.</p>
<p><strong>Details</strong>:</p>
<p>&quot;airflow.cfg&quot; and &quot;airflow_local_settings.py&quot; file needs to exist on the Scheduler (Whether your Scheduler is on a VM or a POD isn't relevant here). We have also added documentation on where to out this file: <a href="https://airflow.apache.org/docs/stable/concepts.html#where-to-put-airflow-local-settings-py" rel="nofollow noreferrer">https://airflow.apache.org/docs/stable/concepts.html#where-to-put-airflow-local-settings-py</a></p>
<p>The <code>pod_mutation_hook</code> now is used whenever you use <code>KubernetesExecutor</code> or <code>KubernetePodOperator</code>. The PODs launched by KubernetesExecutor or KubernetePodOperator will use this mutation hook.</p>
<p>Now, back to the configmap. The case when you are using <code>KubernetesExecutor</code> and have a task that use KuberneretPodOperator, you need both <code>airflow.cfg</code> and <code>airflow_local_settings.py</code> file to exist on the worker pods launched by the KubernetesExecutor.</p>
<p>The KubernetesExecutor launches a Worker Pod for this task.</p>
<p>Scheduler Pod ---&gt; Worker Pod (Pod_1 -- launched by KubernetesExecuetor) --&gt; (Pod_2 -- launched by
Pod_1 the task using KubernetePodOperator)</p>
<p>Now the entire <strong>[kubernetes]</strong> section in airflow.cfg (<a href="https://github.com/apache/airflow/blob/master/airflow/config_templates/default_airflow.cfg#L870-L1028" rel="nofollow noreferrer">https://github.com/apache/airflow/blob/master/airflow/config_templates/default_airflow.cfg#L870-L1028</a>) is used only for KubernetesExecutor and affects what is mounted on the <strong>Worker Pods</strong> launched by KubernetesExecutor.</p>
<p>If you don't specify <code>airflow_local_settings</code> configmap,  the airflow_local_settings file would not be mounted to the worker pod (Pod_1 in the above example) and only airflow.cfg file is mounted. So now for Pod_2 (launched by Pod_1) -- (special case when you use KubernetesPodOperator with KubernetesExecutor), since Pod_1 (the worker POD) doesn't have <code>airflow_local_settings.py</code> file even though the Scheduler has it, Pod_2 would not be mutated since the file doesn't exist over there.</p>
<p>Consider it same as airflow.cfg -- why do you mount <code>airflow.cfg</code> file both to the Scheduler POD and the worker POD. Similarly for this edge case you need <code>airflow_local_settings.py</code> file at both the places.</p>
<p><a href="https://github.com/apache/airflow/blob/ba2d6408e64f219e8f53a20a5a149e3d8109db31/airflow/kubernetes/worker_configuration.py#L279-L305" rel="nofollow noreferrer">https://github.com/apache/airflow/blob/ba2d6408e64f219e8f53a20a5a149e3d8109db31/airflow/kubernetes/worker_configuration.py#L279-L305</a> --&gt; This code is used to decide what is mounted on the Worker Pod (<strong>REF_1</strong>)</p>
<p><a href="https://github.com/apache/airflow/blob/ba2d6408e64f219e8f53a20a5a149e3d8109db31/airflow/executors/kubernetes_executor.py#L462-L481" rel="nofollow noreferrer">https://github.com/apache/airflow/blob/ba2d6408e64f219e8f53a20a5a149e3d8109db31/airflow/executors/kubernetes_executor.py#L462-L481</a> --&gt; Pod created for each task run by KubernetesExecutor (<strong>REF_2</strong>) -- The mutation is applied to this POD as this is launched by the Scheduler and it has <code>airflow_local_settings.py</code> file</p>
<p><a href="https://github.com/apache/airflow/blob/ba2d6408e64f219e8f53a20a5a149e3d8109db31/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py#L383" rel="nofollow noreferrer">https://github.com/apache/airflow/blob/ba2d6408e64f219e8f53a20a5a149e3d8109db31/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py#L383</a> --&gt; This code is used to create a new POD when using KubernetesPod Operator (REF_3) -- Since <code>airflow_local_settings.py</code> wasn't mounted on POD generated in <strong>REF_2</strong> the mutations weren't applied to this POD.</p>
