title: Airbnb Airflow using all system resources
body: <p>We've set up Airbnb/Apache Airflow for our ETL using <code>LocalExecutor</code>, and as we've started building more complex DAGs, we've noticed that Airflow has starting using up incredible amounts of system resources. This is surprising to us because we mostly use Airflow to orchestrate tasks that happen on other servers, so Airflow DAGs spend most of their time waiting for them to complete--there's no actual execution that happens locally.</p>
  
  <p>The biggest issue is that Airflow seems to use up 100% of CPU at all times (on an AWS t2.medium), and uses over 2GB of memory with the default airflow.cfg settings.</p>
  
  <p>If relevant, we're running Airflow using docker-compose running the container twice; once as <code>scheduler</code> and once as <code>webserver</code>.</p>
  
  <p>What are we doing wrong here?  Is this normal?</p>
  
  <p><strong>EDIT:</strong>
  Here is the output from <code>htop</code>, ordered by % Memory used (since that seems to be the main issue now, I got CPU down):
  <a href="https://i.stack.imgur.com/Dq9dh.png" rel="noreferrer"><img src="https://i.stack.imgur.com/Dq9dh.png" alt="Htop"></a>
  <a href="https://i.stack.imgur.com/AnCx6.png" rel="noreferrer"><img src="https://i.stack.imgur.com/AnCx6.png" alt="Htop2"></a></p>
  
  <p>I suppose in theory I could reduce the number of gunicorn workers (it's at the default of 4), but I'm not sure what all the <code>/usr/bin/dockerd</code> processes are. If Docker is complicating things I could remove it, but it's made deployment of changes really easy and I'd rather not remove it if possible.</p>