title: Wait for all the downstream tasks from the previous run in Apache Airflow
body: <p>I am running multiple instances. So,<code> max_active_runs=1</code> is not a solution for me.
  <code>wait_for_downstream</code> can wait only for the immediately downstream task.</p>
  <p>From <code>baseoperator.py</code> code,</p>
  <blockquote>
  <p>wait_for_downstream: when set to true, an instance of task
  X will wait for tasks immediately downstream of the previous instance
  of task X to finish successfully before it runs. This is useful if the
  different instances of a task X alter the same asset, and this asset
  is used by tasks downstream of task X. Note that depends_on_past
  is forced to True wherever wait_for_downstream is used. Also note that
  only tasks <em>immediately</em> downstream of the previous task instance are waited
  for; the statuses of any tasks further downstream are ignored.</p>
  </blockquote>
  <p>How to wait for all the downstream tasks?</p>
  <p>There are multiple parallel steps in my DAG.</p>
  <p><a href="https://i.stack.imgur.com/Gy9Pi.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/Gy9Pi.png" alt="enter image description here" /></a></p>
  <p><strong>EDIT:</strong></p>
  <p>I tried ExternalTaskSensor as suggested by Relic16 to wait for C1/C2/C3 .. to get completed. It's creating a race condition during backfill. As there are multiple instances running at a time though <code>depends_on_past</code> is True. As soon as 1st run's ExternalTaskSensor is in SUCCESS and the actual next task(A1/A2/A3.. ) is processed, it triggers ExternalTaskSensor in the 2nd dag run and it consumers all the slots (as there are more than 16 parallel steps) and keep waiting for C1/C2/C3 .. to get completed.</p>