<p>I have airflow 1.10.5 on a Kubernetes cluster.</p>
<p>The DAGs are written with Kubernetes operator so that they can spin pods for each task inside the DAG on execution, on the k8s cluster.</p>
<p>I have 10 worker nodes.</p>
<p>The pods created by airflow are being created on the same node, where airflow is running.
When many pods have to spin up, they all are queued on the same node, which makes many pod failures due to lack of resources on the node.</p>
<p>At the same time, all other 9 nodes are being used very less, as we have huge load only for the airflow jobs.</p>
<p>How to make the airflow to use all the worker nodes of the k8s cluster?</p>
<p>I do not use any of node affinity or node selector.</p>

tshrinivasan
<p>Solved this little 'issue' by attaching <code>affinity</code> to workers pods manually in Helm chart suggested by the document: <a href="https://airflow.apache.org/docs/helm-chart/stable/parameters-ref.html#workers" rel="nofollow noreferrer">Airflow Helm-Chart </a>.</p>
<pre class="lang-yaml prettyprint-override"><code>workers:
  affinity:
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
        - podAffinityTerm:
            labelSelector:
              matchLabels:
                component: worker
            topologyKey: kubernetes.io/hostname
          weight: 100
</code></pre>
<p>The Airflow Helm Chart <code>values.yaml</code> defines this affinity and says that it will be the default for worker.</p>
<pre class="lang-yaml prettyprint-override"><code>  affinity: {}
  # default worker affinity is:
  #  podAntiAffinity:
  #    preferredDuringSchedulingIgnoredDuringExecution:
  #    - podAffinityTerm:
  #        labelSelector:
  #          matchLabels:
  #            component: worker
  #        topologyKey: kubernetes.io/hostname
  #      weight: 100
</code></pre>
<p>But it fails to mention that this doesn't apply to worker pod but only to worker deployment under <code>CeleryExecutor</code> or <code>CeleryKubernetesExecutor</code>in <code>worker-deployment.yaml</code>.</p>
<pre><code>...
################################
## Airflow Worker Deployment
#################################
{{- $persistence := .Values.workers.persistence.enabled }}
{{- if or (eq .Values.executor &quot;CeleryExecutor&quot;) (eq .Values.executor &quot;CeleryKubernetesExecutor&quot;) }}
...
</code></pre>
<p>So if your do want do spread out your worker pods more, you need to add this affinity(or other custom affinity) to your worker pod template, which can be done through Helm values.yaml.</p>
<p>Though i don't think that this will be considered as a 'issue' as most likely the certain node is free enough so Kubernetes keeps scheduling workers pods to it. When system load goes high, Kubernetes will spread out workers pods. And having worker pods in same node might reduce the network traffic between nodes in some cases.</p>
<p>But in my case when all worker pods are being scheduled to the same nodes, the pods initialization latency is higher than having a distributed workload. So i decide to spread them out across the cluster.</p>
