{"action":"opened","issue":{"url":"https://api.github.com/repos/airbnb/airflow/issues/1023","repository_url":"https://api.github.com/repos/airbnb/airflow","labels_url":"https://api.github.com/repos/airbnb/airflow/issues/1023/labels{/name}","comments_url":"https://api.github.com/repos/airbnb/airflow/issues/1023/comments","events_url":"https://api.github.com/repos/airbnb/airflow/issues/1023/events","html_url":"https://github.com/airbnb/airflow/issues/1023","id":134409320,"number":1023,"title":"all airflow dag runs failed on mesos cluster (mesosExecutor)","user":{"login":"birdandbees","id":2340768,"avatar_url":"https://avatars.githubusercontent.com/u/2340768?v=3","gravatar_id":"","url":"https://api.github.com/users/birdandbees","html_url":"https://github.com/birdandbees","followers_url":"https://api.github.com/users/birdandbees/followers","following_url":"https://api.github.com/users/birdandbees/following{/other_user}","gists_url":"https://api.github.com/users/birdandbees/gists{/gist_id}","starred_url":"https://api.github.com/users/birdandbees/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/birdandbees/subscriptions","organizations_url":"https://api.github.com/users/birdandbees/orgs","repos_url":"https://api.github.com/users/birdandbees/repos","events_url":"https://api.github.com/users/birdandbees/events{/privacy}","received_events_url":"https://api.github.com/users/birdandbees/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2016-02-17T21:42:01Z","updated_at":"2016-02-17T21:42:01Z","closed_at":null,"body":"So I first tested airflow dags on a single-node mesos cluster and it worked fine. When I moved my jobs to a 3-node mesos cluster, all dags failed (airflow included examples and my own simple dag).  Wanted to know if anyone has the same experience or did I miss anything? Thanks,\r\n(IP address is removed)\r\n\r\n\r\nMesos-master log:\r\n```\r\nI0217 20:15:18.857110 19177 master.cpp:4789] Status update TASK_RUNNING (UUID: ac13b339-25a8-4104-85c2-036046db8778) for task 0 of framework 7151a68a-e33e-46fb-8244-a47d1c884abb-0002 from slave 94ac5d8c-2f55-4a0f-8826-85e58407e5a4-S1 at slave(1)@:5051\r\nI0217 20:15:18.857192 19177 master.cpp:4837] Forwarding status update TASK_RUNNING (UUID: ac13b339-25a8-4104-85c2-036046db8778) for task 0 of framework 7151a68a-e33e-46fb-8244-a47d1c884abb-0002\r\nI0217 20:15:18.857280 19177 master.cpp:6445] Updating the state of task 0 of framework 7151a68a-e33e-46fb-8244-a47d1c884abb-0002 (latest state: TASK_RUNNING, status update state: TASK_RUNNING)\r\nI0217 20:15:18.858116 19172 master.cpp:3947] Processing ACKNOWLEDGE call ac13b339-25a8-4104-85c2-036046db8778 for task 0 of framework 7151a68a-e33e-46fb-8244-a47d1c884abb-0002 (Airflow) at scheduler-c2278c59-90de-4cfe-a18c-237f69e61cae@:45113 on slave 94ac5d8c-2f55-4a0f-8826-85e58407e5a4-S1\r\nI0217 20:15:19.259552 19176 master.cpp:4789] Status update TASK_FAILED (UUID: 9406e9db-0387-4011-b84c-1455ae2b5e73) for task 0 of framework 7151a68a-e33e-46fb-8244-a47d1c884abb-0002 from slave 94ac5d8c-2f55-4a0f-8826-85e58407e5a4-S1 at slave(1)@:5051 \r\nI0217 20:15:19.259608 19176 master.cpp:4837] Forwarding status update TASK_FAILED (UUID: 9406e9db-0387-4011-b84c-1455ae2b5e73) for task 0 of framework 7151a68a-e33e-46fb-8244-a47d1c884abb-0002\r\n```\r\nMesos-slave log:\r\n```\r\nI0217 20:15:12.976652 20778 slave.cpp:1862] Sending queued task '0' to executor '0' of framework 7151a68a-e33e-46fb-8244-a47d1c884abb-0002 at executor(1)@:47814\r\nI0217 20:15:12.979185 20782 slave.cpp:3001] Handling status update TASK_RUNNING (UUID: ac13b339-25a8-4104-85c2-036046db8778) for task 0 of framework 7151a68a-e33e-46fb-8244-a47d1c884abb-0002 from executor(1)@:47814\r\nI0217 20:15:12.979318 20782 status_update_manager.cpp:320] Received status update TASK_RUNNING (UUID: ac13b339-25a8-4104-85c2-036046db8778) for task 0 of framework 7151a68a-e33e-46fb-8244-a47d1c884abb-0002\r\nI0217 20:15:12.979502 20782 slave.cpp:3353] Forwarding the update TASK_RUNNING (UUID: ac13b339-25a8-4104-85c2-036046db8778) for task 0 of framework 7151a68a-e33e-46fb-8244-a47d1c884abb-0002 to master@:5050\r\nI0217 20:15:12.979593 20782 slave.cpp:3263] Sending acknowledgement for status update TASK_RUNNING (UUID: ac13b339-25a8-4104-85c2-036046db8778) for task 0 of framework 7151a68a-e33e-46fb-8244-a47d1c884abb-0002 to executor(1)@:47814\r\nI0217 20:15:12.981434 20782 status_update_manager.cpp:392] Received status update acknowledgement (UUID: ac13b339-25a8-4104-85c2-036046db8778) for task 0 of framework 7151a68a-e33e-46fb-8244-a47d1c884abb-0002\r\nI0217 20:15:13.381697 20778 slave.cpp:3001] Handling status update TASK_FAILED (UUID: 9406e9db-0387-4011-b84c-1455ae2b5e73) for task 0 of framework 7151a68a-e33e-46fb-8244-a47d1c884abb-0002 from executor(1)@:47814\r\nI0217 20:15:13.382037 20781 status_update_manager.cpp:320] Received status update TASK_FAILED (UUID: 9406e9db-0387-4011-b84c-1455ae2b5e73) for task 0 of framework 7151a68a-e33e-46fb-8244-a47d1c884abb-0002\r\n```\r\n\r\nairflow dag task log: \r\n```\r\n[2016-02-17 20:15:18,804] {mesos_executor.py:52} INFO - AirflowScheduler registered to mesos with framework ID 7151a68a-e33e-46fb-8244-a47d1c884abb-0002\r\n[2016-02-17 20:15:18,804] {mesos_executor.py:105} INFO - Received offer 7151a68a-e33e-46fb-8244-a47d1c884abb-O4 with cpus: 4.0 and mem: 15023.0\r\n[2016-02-17 20:15:18,805] {mesos_executor.py:118} INFO - Launching task 0 using offer 7151a68a-e33e-46fb-8244-a47d1c884abb-O4\r\n[2016-02-17 20:15:18,805] {mesos_executor.py:105} INFO - Received offer 7151a68a-e33e-46fb-8244-a47d1c884abb-O5 with cpus: 4.0 and mem: 15023.0\r\n[2016-02-17 20:15:18,857] {mesos_executor.py:149} INFO - Task 0 is in state TASK_RUNNING, data\r\n[2016-02-17 20:15:19,260] {mesos_executor.py:149} INFO - Task 0 is in state TASK_FAILED, data\r\n                                                       \r\n```\r\n"}}