<p>Like some article that I previously read. It said that in new Kubernetes version, already include Spark capabilities. But with some different ways such as using KubernetesPodOperator instead of using BashOperator / PythonOperator to do SparkSubmit.</p>

<p>Is that the best practice to Combine Airflow + Kubernetes is to remove Spark and using KubernetesPodOperator to execute the task?</p>

<p>Which is have a better performance since Kubernetes have AutoScaling that Spark doesn’t have.</p>

<p>Need someone expert in Kubernetes to help me explain this. I’m still newbie with this Kubernetes, Spark, and Airflow things. :slight_smile:</p>

<p>Thank You.</p>

Xnuxer
<blockquote>
  <p>in new Kubernetes version, already include Spark capabilities</p>
</blockquote>

<p>I think you got that backwards. New versions of Spark can run tasks in a Kubernetes cluster.</p>

<blockquote>
  <p>using KubernetesPodOperator instead of using BashOperator / PythonOperator to do SparkSubmit</p>
</blockquote>

<p>Using Kubernetes would allow you to run containers with whatever isolated dependencies you wanted. </p>

<p>Meaning </p>

<ol>
<li>With BashOperator, you must distribute the files to some shared filesystem or to all the nodes that ran the Airflow tasks. For example, <code>spark-submit</code> must be available on all Airflow nodes. </li>
<li>Similarly with Python, you ship out some zip or egg files that include your pip/conda dependency environment</li>
</ol>

<blockquote>
  <p>remove Spark and using KubernetesPodOperator to execute the task</p>
</blockquote>

<p>There is still good reasons to run Spark with Airflow, but instead you would be packaging a Spark driver container to execute <code>spark-submit</code> inside a container against the Kubernetes cluster. This way, you only need <code>docker</code> installed, not Spark (and all dependencies)</p>

<blockquote>
  <p>Kubernetes have AutoScaling that Spark doesn’t have</p>
</blockquote>

<p>Spark does have <a href="https://spark.apache.org/docs/latest/job-scheduling.html#dynamic-resource-allocation" rel="nofollow noreferrer">Dynamic Resource Allocation...</a></p>
