{"action":"created","issue":{"url":"https://api.github.com/repos/apache/incubator-airflow/issues/1707","repository_url":"https://api.github.com/repos/apache/incubator-airflow","labels_url":"https://api.github.com/repos/apache/incubator-airflow/issues/1707/labels{/name}","comments_url":"https://api.github.com/repos/apache/incubator-airflow/issues/1707/comments","events_url":"https://api.github.com/repos/apache/incubator-airflow/issues/1707/events","html_url":"https://github.com/apache/incubator-airflow/pull/1707","id":169479766,"number":1707,"title":"[AIRFLOW-398] Option to use cgroups to limit task resource utilization","user":{"login":"plypaul","id":5704934,"avatar_url":"https://avatars.githubusercontent.com/u/5704934?v=3","gravatar_id":"","url":"https://api.github.com/users/plypaul","html_url":"https://github.com/plypaul","followers_url":"https://api.github.com/users/plypaul/followers","following_url":"https://api.github.com/users/plypaul/following{/other_user}","gists_url":"https://api.github.com/users/plypaul/gists{/gist_id}","starred_url":"https://api.github.com/users/plypaul/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/plypaul/subscriptions","organizations_url":"https://api.github.com/users/plypaul/orgs","repos_url":"https://api.github.com/users/plypaul/repos","events_url":"https://api.github.com/users/plypaul/events{/privacy}","received_events_url":"https://api.github.com/users/plypaul/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2016-08-04T21:38:35Z","updated_at":"2016-08-05T18:19:29Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/incubator-airflow/pulls/1707","html_url":"https://github.com/apache/incubator-airflow/pull/1707","diff_url":"https://github.com/apache/incubator-airflow/pull/1707.diff","patch_url":"https://github.com/apache/incubator-airflow/pull/1707.patch"},"body":"Dear Airflow Maintainers,\r\n\r\nPlease accept this PR that addresses the following issues:\r\n- https://issues.apache.org/jira/browse/AIRFLOW-398\r\n\r\nCurrently, this is getting added to `contrib`.\r\n\r\nTesting Done:\r\n- Tested locally, but unit tests not included due to issues with setting cgroups inside Travis. Looking into this further...\r\n- Sample run where a high-memory task is killed due to OOM\r\n```\r\n[2016-08-04 21:28:22,868] {models.py:162} INFO - Filling up the DagBag from /home/paul_yang/airflow/dags/test_dag.py\r\n[2016-08-04 21:28:23,014] {cgroup_task_runner.py:52} INFO - Creating cgroup 5f2bdc08-5a8a-11e6-99a5-22000b8c08f9 in /memory/airflow/2016-08-04\r\n[2016-08-04 21:28:23,014] {cgroup_task_runner.py:102} INFO - Setting memory/airflow/2016-08-04/5f2bdc08-5a8a-11e6-99a5-22000b8c08f9 with 512 MB of memory\r\n[2016-08-04 21:28:23,020] {cgroup_task_runner.py:52} INFO - Creating cgroup 5f2bdc08-5a8a-11e6-99a5-22000b8c08f9 in /cpu/airflow/2016-08-04\r\n[2016-08-04 21:28:23,021] {cgroup_task_runner.py:109} INFO - Setting cpu/airflow/2016-08-04/5f2bdc08-5a8a-11e6-99a5-22000b8c08f9 with 1024 CPU shares\r\n[2016-08-04 21:28:23,021] {cgroup_task_runner.py:113} INFO - Starting task process with cgroups cpu,memory:airflow/2016-08-04/5f2bdc08-5a8a-11e6-99a5-22000b8c08f9\r\n[2016-08-04 21:28:23,021] {cgroup_task_runner.py:119} INFO - Running: ['cgexec', '-g', 'cpu,memory:airflow/2016-08-04/5f2bdc08-5a8a-11e6-99a5-22000b8c08f9', u'airflow', u'run', 'test_dag', 'oom', '2016-01-01T00:00:00', u'--job_id', '248', u'--force', u'--raw', u'-sd', u'DAGS_FOLDER/test_dag.py']\r\n[2016-08-04 21:28:23,753] {models.py:162} INFO - Filling up the DagBag from /home/paul_yang/airflow/dags/test_dag.py\r\n[2016-08-04 21:28:23,801] {models.py:1320} INFO -\r\n--------------------------------------------------------------------------------\r\nStarting attempt 1 of 2\r\n--------------------------------------------------------------------------------\r\n\r\n[2016-08-04 21:28:23,828] {models.py:1343} INFO - Executing <Task(BashOperator): oom> on 2016-01-01 00:00:00\r\n[2016-08-04 21:28:23,846] {bash_operator.py:69} INFO - tmp dir root location:\r\n/tmp\r\n[2016-08-04 21:28:23,847] {bash_operator.py:78} INFO - Temporary script location :/tmp/airflowtmpZhB3HD//tmp/airflowtmpZhB3HD/oomo1Vb1L\r\n[2016-08-04 21:28:23,847] {bash_operator.py:79} INFO - Running command: python /home/paul_yang/python/mem.py\r\n[2016-08-04 21:28:23,850] {bash_operator.py:87} INFO - Output:\r\n[2016-08-04 21:28:24,100] {bash_operator.py:91} INFO - /tmp/airflowtmpZhB3HD/oomo1Vb1L: line 1: 108392 Killed                  python /home/paul_yang/python/mem.py\r\n[2016-08-04 21:28:24,101] {bash_operator.py:94} INFO - Command exited with return code 137\r\n[2016-08-04 21:28:24,101] {models.py:1410} ERROR - Bash command failed\r\nTraceback (most recent call last):\r\n  File \"/mnt/home/paul_yang/forked/airflow/airflow/models.py\", line 1369, in run\r\n    result = task_copy.execute(context=context)\r\n  File \"/mnt/home/paul_yang/forked/airflow/airflow/operators/bash_operator.py\", line 97, in execute\r\n    raise AirflowException(\"Bash command failed\")\r\nAirflowException: Bash command failed\r\n[2016-08-04 21:28:24,102] {models.py:1422} INFO - Marking task as UP_FOR_RETRY\r\n[2016-08-04 21:28:24,121] {models.py:1451} ERROR - Bash command failed\r\n[2016-08-04 21:28:28,032] {cgroup_task_runner.py:80} INFO - Deleting cgroup <Node /memory/airflow/2016-08-04>/5f2bdc08-5a8a-11e6-99a5-22000b8c08f9\r\n[2016-08-04 21:28:28,038] {cgroup_task_runner.py:80} INFO - Deleting cgroup <Node /cpu/airflow/2016-08-04>/5f2bdc08-5a8a-11e6-99a5-22000b8c08f9\r\n```\r\n\r\nTo prevent Airflow tasks from using too much memory and causing machines\r\nto go OOM, this introduces an optional mechanism to use cgroups to\r\nlimit task resource utilization."},"comment":{"url":"https://api.github.com/repos/apache/incubator-airflow/issues/comments/237924607","html_url":"https://github.com/apache/incubator-airflow/pull/1707#issuecomment-237924607","issue_url":"https://api.github.com/repos/apache/incubator-airflow/issues/1707","id":237924607,"user":{"login":"plypaul","id":5704934,"avatar_url":"https://avatars.githubusercontent.com/u/5704934?v=3","gravatar_id":"","url":"https://api.github.com/users/plypaul","html_url":"https://github.com/plypaul","followers_url":"https://api.github.com/users/plypaul/followers","following_url":"https://api.github.com/users/plypaul/following{/other_user}","gists_url":"https://api.github.com/users/plypaul/gists{/gist_id}","starred_url":"https://api.github.com/users/plypaul/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/plypaul/subscriptions","organizations_url":"https://api.github.com/users/plypaul/orgs","repos_url":"https://api.github.com/users/plypaul/repos","events_url":"https://api.github.com/users/plypaul/events{/privacy}","received_events_url":"https://api.github.com/users/plypaul/received_events","type":"User","site_admin":false},"created_at":"2016-08-05T18:19:29Z","updated_at":"2016-08-05T18:19:29Z","body":"Thanks @bolkedebruin! Let me get some documentation written regarding this..."}}
{"actor":{"display_login":"plypaul"}}