title: Composer/Airflow struggles running Dataflow jobs on preemptible VMs
body: <p>I have a Composer/Airflow instance in GCP (version composer-1.12.2-airflow-1.10.10), which runs its current jobs acceptably (approx. 40 DAGs with 6 Dataflow jobs each and 18 other shorter tasks in each DAG).
  The Dataflow jobs are run from a Dataflow template (normal template, not flex template) and typically take around 20min to complete, partly in parallel. I start them with a python <a href="https://github.com/apache/airflow/blob/e03a3f456fe087d92a1d863a0cbc56d1d65b091c/airflow/providers/google/cloud/operators/dataflow.py#L280" rel="nofollow noreferrer">DataflowTemplatedJobStartOperator</a></p>
  <p>If I use the Dataflow option <a href="https://cloud.google.com/dataflow/docs/guides/flexrs" rel="nofollow noreferrer">--flexRSGoal=COST_OPTIMIZED</a>, the instance clogs up. At best it manages to schedule one Dataflow job per minute to start with over the first 20min, at high CPU usage. This results in tasks accumulating, slowing it down even more until it almost stops scheduling.</p>
  <p>The flexRSGoal setting is the only difference between the working and problematic setup.</p>
  <p>I expect the DataflowTemplatedJobStartOperator doesn't properly support the situation where Dataflow jobs are in state &quot;Queued&quot; for a while before they start - or that I need to set/adjust some other parameters to make this work. Does anybody have an idea? Thank you.</p>