<p>I'm trying to set up a DAG that will create a Spark Cluster in the first task, submit spark applications to the cluster in interim tasks, and have finally teardown the Spark Cluster in the last task.</p>

<p>The approach I'm attempting right now is to use KubernetesPodOperators to create Spark Master and Worker pods. The issue is that they run a spark daemon which never exits. The fact that the command called on the pod never exits means that those tasks gets stuck in airflow in a running phase. So, I'm wondering if there's a way to run the spark daemon and then continue on to the next tasks in the DAG?</p>

sstannus
<blockquote>
  <p>The approach I'm attempting right now is to use KubernetesPodOperators to create Spark Master and Worker pods.</p>
</blockquote>

<p>Apache Spark <a href="https://spark.apache.org/docs/latest/running-on-kubernetes.html" rel="nofollow noreferrer">provides working support</a> for executing jobs in a Kubernetes cluster. It delivers a driver that is capable of starting executors in pods to run jobs. </p>

<p>You don't need to create <em>Master</em> and <em>Worker</em> pods directly in Airflow.</p>

<p>Rather build a <a href="https://spark.apache.org/docs/latest/running-on-kubernetes.html#docker-images" rel="nofollow noreferrer">Docker image containing Apache Spark with Kubernetes backend</a>.
An <a href="https://github.com/apache/spark/blob/master/resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile" rel="nofollow noreferrer">example Dockerfile</a> is provided in the project.</p>

<p>Then <a href="https://spark.apache.org/docs/latest/running-on-kubernetes.html#cluster-mode" rel="nofollow noreferrer">submit the given jobs</a> to the cluster in a container based off this image by using <code>KubernetesPodOperator</code>. The following sample job is adapted from documentation provided in Apache Spark to submit spark jobs directly to a Kubernetes cluster.</p>

<pre class="lang-py prettyprint-override"><code>from airflow.operators.kubernetes_pod_operator import KubernetesPodOperator


kubernetes_full_pod = KubernetesPodOperator(
    task_id='spark-job-task-ex',
    name='spark-job-task',
    namespace='default',
    image='&lt;prebuilt-spark-image-name&gt;',
    cmds=['bin/spark-submit'],
    arguments=[
        '--master k8s://https://&lt;k8s-apiserver-host&gt;:&lt;k8s-apiserver-port&gt;',
        '--deploy-mode cluster',
        '--name spark-pi',
        ' --class org.apache.spark.examples.SparkPi',
        '--conf spark.executor.instances=5',
        '--conf spark.kubernetes.container.image=&lt;prebuilt-spark-image-name&gt;',
        'local:///path/to/examples.jar'
    ],
    #...
)
</code></pre>
