<p>I am evaluating Apache airflow for production use in a data environment and I would like to know if with airflow you can run operators in self contained docker environments on an auto scaling Kubernetes cluster.</p>

<p>I found the following operator: <code>KubernetesPodOperator</code> which seems to do the job, but the only examples I have found have been on Google Cloud. I would like to run this on AWS, however I haven't found any examples of how this would be done. I believe AWS EKS or AWS fargate might fit the bill but not sure.</p>

<p>Can anyone with airflow experience please let me know if this is possible? I have looked online and haven't found anything clear yet. </p>

maldman
<p>We have been using Fargate and Airflow in production and the experience so far has been good. </p>

<p>We have been using it for transient workloads and it is turning out to be cheaper for us than having a dedicated Kubernetes cluster. Also, there is no management overhead of any kind.</p>

<p><a href="https://github.com/ishan4488/airflow-fargate-example/tree/master/airflow-fargate-example" rel="nofollow noreferrer">Github — Airflow DAG with ECSOperatorConfig</a></p>
