<p>We have Airflow working with KubernetesExecutor.</p>
<p>And it takes a lot of time to just start a new task on schedule.
That's a usual state when a couple of dags must be started at same time, but a lot of tasks are in state &quot;queued&quot;:</p>
<p><a href="https://i.stack.imgur.com/6cvFa.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/6cvFa.png" alt="Tasks queue" /></a></p>
<p>It looks like we have not enough workers, but our devops says that there is a separate worker for each dag task.</p>
<p>On the image below we can see that total DAG execution time is 1:57:</p>
<p><a href="https://i.stack.imgur.com/xa2tD.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/xa2tD.png" alt="DAG execution duration" /></a></p>
<p>But each task execution time in sum is at maximum 3 seconds.</p>
<p><a href="https://i.stack.imgur.com/uysbB.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/uysbB.png" alt="enter image description here" /></a>
<a href="https://i.stack.imgur.com/AneIK.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/AneIK.png" alt="enter image description here" /></a>
<a href="https://i.stack.imgur.com/AmwpY.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/AmwpY.png" alt="enter image description here" /></a></p>
<p>Any ideas why it takes so much time to just start a task?
I think that there is some problem with Kubernetes or its configuration, but have no proofs and have no competency to prove it.</p>
<p>P.S. I am not from DevOps team, so unfortunatelly I do not have access to server or kubernetes configuraion, but I can ask for it on demand.</p>
<p>P.P.S On previous server with LocalExecutor everything worked as expected, without delays.</p>

Evgenii Kozlov
<p>The Kubernetes executor spins up a new pod for each task, which includes the startup time for Airflow's worker process. During that startup time, the task will be in the &quot;queued&quot; state. Depending on your Kubernetes setup and the nature of the worker pod you are trying to spin up, start up times of 2 minutes are not unheard of. Your best bet to determine what's causing the time is to watch it on K8s with your ops team, it may be in the Pending state or it could be Running but going through the Airflow start up.</p>
<p>The LocalExecutor, by contrast has none of that start up time, so it makes sense that tasks start faster on the LocalExecutor.</p>
