{"action":"created","issue":{"url":"https://api.github.com/repos/airbnb/airflow/issues/1374","repository_url":"https://api.github.com/repos/airbnb/airflow","labels_url":"https://api.github.com/repos/airbnb/airflow/issues/1374/labels{/name}","comments_url":"https://api.github.com/repos/airbnb/airflow/issues/1374/comments","events_url":"https://api.github.com/repos/airbnb/airflow/issues/1374/events","html_url":"https://github.com/airbnb/airflow/pull/1374","id":148176109,"number":1374,"title":"Fix logic in Airflow scheduler to prevent DAGs from being reloaded on…","user":{"login":"criccomini","id":167569,"avatar_url":"https://avatars.githubusercontent.com/u/167569?v=3","gravatar_id":"","url":"https://api.github.com/users/criccomini","html_url":"https://github.com/criccomini","followers_url":"https://api.github.com/users/criccomini/followers","following_url":"https://api.github.com/users/criccomini/following{/other_user}","gists_url":"https://api.github.com/users/criccomini/gists{/gist_id}","starred_url":"https://api.github.com/users/criccomini/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/criccomini/subscriptions","organizations_url":"https://api.github.com/users/criccomini/orgs","repos_url":"https://api.github.com/users/criccomini/repos","events_url":"https://api.github.com/users/criccomini/events{/privacy}","received_events_url":"https://api.github.com/users/criccomini/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":6,"created_at":"2016-04-13T20:33:50Z","updated_at":"2016-04-13T22:49:17Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/airbnb/airflow/pulls/1374","html_url":"https://github.com/airbnb/airflow/pull/1374","diff_url":"https://github.com/airbnb/airflow/pull/1374.diff","patch_url":"https://github.com/airbnb/airflow/pull/1374.patch"},"body":"I have been observing strange behavior from the Airflow scheduler lately. The behavior manifests itself as having a lot of active DAG runs with very few task instances being executed. Then, every two minutes, or so, a bunch of task instances would get scheduled. They'd then finish up pretty quickly (< 30s), and there would be no activity for 1m30s until more tasks were added.\r\n\r\nUpon investigation, it appears that the Airflow scheduler was taking more than 2 minutes to complete a single loop. We put in profiling in the jobs.py _execute method, and it became apparent that this loop was the culprit:\r\n\r\n                for dag in dags:\r\n                    self.logger.debug(\"Scheduling {}\".format(dag.dag_id))\r\n                    dag = dagbag.get_dag(dag.dag_id)\r\n                    if not dag or (dag.dag_id in paused_dag_ids):\r\n                        continue\r\n                    try:\r\n                        self.schedule_dag(dag)\r\n                        self.process_dag(dag, executor)\r\n                        self.manage_slas(dag)\r\n                    except Exception as e:\r\n                        self.logger.exception(e)\r\n\r\nThe dagbag.get_dag(dag.dag_id) invocation was taking about 500ms per-DAG. I have 270 DAGs. 270 * .5 = ~2 minutes.\r\n\r\nThe get_dag method has an if-statement to determine if the DAG should be refreshed. Every DAG was getting refreshed on every iteration. This was happening because orm_dag.last_expired is None for all DAGs, which resulted in a check of `dag.last_loaded < datetime(2100, 1, 1)`, which it always is.\r\n\r\nThe fix appears to only check if last_expired is not None. If it's None, the UI hasn't been clicked, and we shouldn't bother checking if it needs to be refreshed (unless it's not in self.dags).\r\n\r\nThis dropped the scheduler loop from ~2 minutes to 16 seconds. This is still not great, but it's better. Now, each iteration of the for loop takes .05s. Still, .05*270 = 13.5 seconds. Ideally, I'd like this loop to be even lower. 270 DAGs aren't *that* many.\r\n\r\nI think the scheduler's loop might need to be re-thought a bit, but I'll leave that for a different issue."},"comment":{"url":"https://api.github.com/repos/airbnb/airflow/issues/comments/209680678","html_url":"https://github.com/airbnb/airflow/pull/1374#issuecomment-209680678","issue_url":"https://api.github.com/repos/airbnb/airflow/issues/1374","id":209680678,"user":{"login":"jlowin","id":153965,"avatar_url":"https://avatars.githubusercontent.com/u/153965?v=3","gravatar_id":"","url":"https://api.github.com/users/jlowin","html_url":"https://github.com/jlowin","followers_url":"https://api.github.com/users/jlowin/followers","following_url":"https://api.github.com/users/jlowin/following{/other_user}","gists_url":"https://api.github.com/users/jlowin/gists{/gist_id}","starred_url":"https://api.github.com/users/jlowin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jlowin/subscriptions","organizations_url":"https://api.github.com/users/jlowin/orgs","repos_url":"https://api.github.com/users/jlowin/repos","events_url":"https://api.github.com/users/jlowin/events{/privacy}","received_events_url":"https://api.github.com/users/jlowin/received_events","type":"User","site_admin":false},"created_at":"2016-04-13T22:49:17Z","updated_at":"2016-04-13T22:49:17Z","body":"The fix makes sense to me!"}}