title: Apache Airflow - how many tasks in a DAG is too many?
body: <p>I tried having a DAG with 400 tasks (like each one running calling remote spark server to process a separate data file into s3...nothing to do with mysql) and airflow (v1.10.3) did the following for the next 15mins:</p>
  
  <ul>
  <li>cpu stayed at 99%</li>
  <li>did not handle new putty login or ssh requests to
  my machine (amazon linux)</li>
  <li>airflow webserver stopped responding..only
  gave 504 errors </li>
  <li>Started 130 concurrent connections to mysql RDS
  (airflow metadb) </li>
  <li>kept my tasks stuck in scheduled state</li>
  </ul>
  
  <p>i eventually switched to another ec2 instance but got same outcome...</p>
  
  <p>I am running LocalExecutor on single machine (16 CPUs).</p>
  
  <p>Note for a DAG with 30 tasks it runs fine.</p>