<p>I have a few DAGs that only run for a few hours a day, after which the Cloud Composer cluster does nothing. I'm trying to understand the worker pool auto-scaling mechanism:</p>

<ul>
<li>If I set the cluster size to 3 nodes, but autoscale with min=1, max=10, then set a DAG's <code>concurrency=10</code>, will Composer run on those 3 nodes until the DAG hints at it that it needs for resources, at which point it auto scales to 10 nodes? Then will it destroy those additional 7 nodes when the DAG is done?</li>
</ul>

rodrigo-silveira
<p>I have decided to elaborate little bit more to @SNeumann answer. As for now, Composer doesn't support Kubernetes clusters with autoscaling enabled. This is due to there's a complex conversation around Kuberenetes resource requests and how those can't be estimated for Airflow pods at this time. </p>

<p>Feature requests have already been filed: <a href="https://issuetracker.google.com/issues/116455284" rel="nofollow noreferrer">1</a>, <a href="https://issuetracker.google.com/issues/117119946" rel="nofollow noreferrer">2</a> to support worker autoscaling based on DAG demand. I suggest <a href="https://developers.google.com/issue-tracker/guides/subscribe#starring_an_issue" rel="nofollow noreferrer">subscribing it</a> to keep updated as you'll receive e-mails regarding its progress. </p>

<p>According to the <a href="https://cloud.google.com/composer/docs/how-to/using/using-kubernetes-pod-operator#environment-resources" rel="nofollow noreferrer">official documentation</a>:</p>

<blockquote>
  <p>the Airflow scheduler and workers are in the same GKE cluster, the
  schedulers and workers won't work properly if the competition results
  in resource starvation.</p>
</blockquote>

<p>For now, you need to enable autoscaling on node level and write .yaml, then update <code>minReplicas: 1</code> and <code>maxReplicas: 50</code> like in the Medium tutorial posted by @SNeumann.</p>

<p>I hope you find the above pieces of information useful.</p>
