<p>I have a few DAGs that only run for a few hours a day, after which the Cloud Composer cluster does nothing. I'm trying to understand the worker pool auto-scaling mechanism:</p>

<ul>
<li>If I set the cluster size to 3 nodes, but autoscale with min=1, max=10, then set a DAG's <code>concurrency=10</code>, will Composer run on those 3 nodes until the DAG hints at it that it needs for resources, at which point it auto scales to 10 nodes? Then will it destroy those additional 7 nodes when the DAG is done?</li>
</ul>

rodrigo-silveira
<p>Cloud Composer doesn't natively support auto-scaling, but you can hack it by changing airflow config and change the underlying k8s settings. </p>

<p>There's a great article on how to do it here: <a href="https://medium.com/traveloka-engineering/enabling-autoscaling-in-google-cloud-composer-ac84d3ddd60" rel="nofollow noreferrer">https://medium.com/traveloka-engineering/enabling-autoscaling-in-google-cloud-composer-ac84d3ddd60</a></p>
