{"action":"closed","issue":{"url":"https://api.github.com/repos/airbnb/airflow/issues/1192","repository_url":"https://api.github.com/repos/airbnb/airflow","labels_url":"https://api.github.com/repos/airbnb/airflow/issues/1192/labels{/name}","comments_url":"https://api.github.com/repos/airbnb/airflow/issues/1192/comments","events_url":"https://api.github.com/repos/airbnb/airflow/issues/1192/events","html_url":"https://github.com/airbnb/airflow/issues/1192","id":142698379,"number":1192,"title":"One dag_run per DAG per second limitation","user":{"login":"xavierp","id":900223,"avatar_url":"https://avatars.githubusercontent.com/u/900223?v=3","gravatar_id":"","url":"https://api.github.com/users/xavierp","html_url":"https://github.com/xavierp","followers_url":"https://api.github.com/users/xavierp/followers","following_url":"https://api.github.com/users/xavierp/following{/other_user}","gists_url":"https://api.github.com/users/xavierp/gists{/gist_id}","starred_url":"https://api.github.com/users/xavierp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/xavierp/subscriptions","organizations_url":"https://api.github.com/users/xavierp/orgs","repos_url":"https://api.github.com/users/xavierp/repos","events_url":"https://api.github.com/users/xavierp/events{/privacy}","received_events_url":"https://api.github.com/users/xavierp/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2016-03-22T16:08:47Z","updated_at":"2016-05-02T01:20:28Z","closed_at":"2016-05-02T01:20:28Z","body":"Hello there, i have decided to open an issue to talk about the one `dag_run` per DAG per seconde limitation.\r\n\r\nHere is the story : i was trying to figure out the best way to handle the parsing of an unknown number of files. For the background, at my company, we are migrating to a new infrastructure and we need to import old logs file. They are not timely based an can be thousands...\r\n\r\nAfter a few tries i asked for help on the [Airflow Google Group](https://groups.google.com/forum/#!topic/airbnb_airflow/eI9eziXZ-W0) and discussed a bit about it with  @artwr .\r\n\r\nTo work this out i made a `TriggerMultipleDagRunOperator` based on the `TriggerDagRunOperator`. The callable function just return a list of `DagRun` objects. Sample code [here](https://gist.github.com/xavierp/e56a656aec0db9ecae8a). (This is far from perfection and was only done for test purposes)\r\n\r\nI was first limited by the unique constraint key (`dag_id`, `execution_date`) on the `dag_run` table.\r\n\r\nSo i wrote a little patch to change that, but after a few tests i realized that a big part of airflow\r\nis based on the fact that only one `dag_run` for a specified `dag_id` should append at the same time.\r\nFor example the webserver interfaces  /graph, /gant, /log are build using only the execution date and not the `dag_id`.\r\n\r\nI think that we should be able to have multiple `dag_run` at the time for the same `dag_id` if they have\r\ndifferent input parameters (especially the `conf` field in the `dag_run` table)\r\nFor example if i have a DAG that do some tasks to import and parse a file, why could i not have multiple `dag_run` at the same time if they have different target file in the `dag_run`.`conf` field ?\r\n\r\nMy questions are :\r\n- Is that the correct and intended behaviour ?\r\n- Do you have a better workout around this ? (For the moment i use my `TriggerMultipleDagRunOperator` \r\nwith a `time.sleep(1)` in the for loop...)\r\n- What do you think about it ?\r\n- Do you plan / are you open to modifications of this ?\r\n\r\nHere is a proposal :\r\nWe could, for example, create a unique key for the `dag_run` based on  (`dag_id`,  `execution_date`, `conf_hash`).\r\nThis would implies some modifications in the database model, in the main Airflow code and in the UI.\r\n\r\nBy the way thanks for the amazing work you have already done on Airflow and i would be very glad to help. :)\r\n\r\nBest regards,\r\n\r\nXavier."}}