<p>I have multiple dags using Celery Executor but I want one particular dag to run using Kubernetes Executor. I am unable to deduce a good and reliable way to achieve this.</p>

<p>I have an <code>airflow.cfg</code> in which I have declared <code>CeleryExecutor</code> to be used. And I don't want to change it since it is really needed in all the dags but one.</p>

<pre><code># The executor class that airflow should use. Choices include
# SequentialExecutor, LocalExecutor, CeleryExecutor
executor = CeleryExecutor
</code></pre>

<p>My dag code:</p>

<pre><code>from datetime import datetime, timedelta

from airflow import DAG
from airflow.contrib.operators.kubernetes_pod_operator import \
    KubernetesPodOperator
from airflow.operators.dummy_operator import DummyOperator

default_args = {
    'owner': 'airflow',
    'depends_on_past': False,
    'start_date': datetime.utcnow(),
    'email': ['airflow@example.com'],
    'email_on_failure': False,
    'email_on_retry': False,
    'retries': 1,
    'retry_delay': timedelta(minutes=5)
}

dag = DAG(
    'kubernetes_sample_1', default_args=default_args)


start = DummyOperator(task_id='run_this_first', dag=dag)

passing = KubernetesPodOperator(namespace='default',
                                image="Python:3.6",
                                cmds=["Python", "-c"],
                                arguments=["print('hello world')"],
                                labels={"foo": "bar"},
                                name="passing-test",
                                task_id="passing-task",
                                get_logs=True,
                                dag=dag
                                )

failing = KubernetesPodOperator(namespace='default',
                                image="ubuntu:1604",
                                cmds=["Python", "-c"],
                                arguments=["print('hello world')"],
                                labels={"foo": "bar"},
                                name="fail",
                                task_id="failing-task",
                                get_logs=True,
                                dag=dag
                                )

passing.set_upstream(start)
failing.set_upstream(start)

</code></pre>

<p>I can put an if-else condition and then change the value from the point where Airflow picks up the configuration. If this sounds right, please tell me the paths and the files. Although I was hoping to get a more mature method, if it exists.</p>

Aviral Srivastava
<p>Now there is the <a href="https://airflow.apache.org/docs/apache-airflow/stable/executor/celery_kubernetes.html" rel="nofollow noreferrer">CeleryKubernetesExecutor</a> (can't see when it was exactly introduced), which requires to set up Celery and Kubernetes up, but also offers the functionalities from both.</p>
<p>In the official documentation, they offer a rule of thumb to decide when it's worth using it:</p>
<blockquote>
<p>We recommend considering the CeleryKubernetesExecutor when your use
case meets:</p>
<p>The number of tasks needed to be scheduled at the peak exceeds the
scale that your Kubernetes cluster can comfortably handle</p>
<p>A relative small portion of your tasks requires runtime isolation.</p>
<p>You have plenty of small tasks that can be executed on Celery workers
but you also have resource-hungry tasks that will be better to run in
predefined environments.</p>
</blockquote>
