{"action":"created","issue":{"url":"https://api.github.com/repos/celery/celery/issues/2080","repository_url":"https://api.github.com/repos/celery/celery","labels_url":"https://api.github.com/repos/celery/celery/issues/2080/labels{/name}","comments_url":"https://api.github.com/repos/celery/celery/issues/2080/comments","events_url":"https://api.github.com/repos/celery/celery/issues/2080/events","html_url":"https://github.com/celery/celery/issues/2080","id":35164016,"number":2080,"title":"Celery worker hangs.","user":{"login":"cameronmaske","id":1007064,"avatar_url":"https://avatars.githubusercontent.com/u/1007064?v=3","gravatar_id":"","url":"https://api.github.com/users/cameronmaske","html_url":"https://github.com/cameronmaske","followers_url":"https://api.github.com/users/cameronmaske/followers","following_url":"https://api.github.com/users/cameronmaske/following{/other_user}","gists_url":"https://api.github.com/users/cameronmaske/gists{/gist_id}","starred_url":"https://api.github.com/users/cameronmaske/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cameronmaske/subscriptions","organizations_url":"https://api.github.com/users/cameronmaske/orgs","repos_url":"https://api.github.com/users/cameronmaske/repos","events_url":"https://api.github.com/users/cameronmaske/events{/privacy}","received_events_url":"https://api.github.com/users/cameronmaske/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/celery/celery/labels/Issue%20Type:%20Bug","name":"Issue Type: Bug","color":"e11d21"},{"url":"https://api.github.com/repos/celery/celery/labels/Worker%20Hangs","name":"Worker Hangs","color":"E60026"}],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":10,"created_at":"2014-06-06T17:08:09Z","updated_at":"2016-06-23T23:41:43Z","closed_at":"2016-06-23T23:41:43Z","body":"We are encountering an odd issue where a worker hangs. Tasks are present in our RabbitMQ queue, but the worker fails to pick them up and process them (IDLE status). Running `inspect active` reveals there are no active tasks on the worker. \r\n \r\nWe are running on 3 EC2 instances, with Ubuntu `12.04`. 1 instance is our celery beat, the other two are workers . \r\nWe are running `celery==3.1.11` and `librabbitmq==1.5.2` and our broker is `RabbitMQ 3.2.4`.  We are also using `Django 1.6.3`.\r\n\r\nHere are our celery settings.\r\n```\r\nCELERYD_PREFETCH_MULTIPLIER = 1\r\nCELERY_TIMEZONE = 'US/Eastern'\r\nCELERYD_HIJACK_ROOT_LOGGER = False\r\nCELERYD_MAX_TASKS_PER_CHILD = 1000\r\nCELERY_IGNORE_RESULT = True\r\nCELERYD_TASK_SOFT_TIME_LIMIT = 600\r\n\r\nBROKER_POOL_LIMIT = 10\r\n```\r\n\r\nFor each queue worker (we have about ~25) we manage them through supervisor. \r\nTaking the form of this...\r\n```\r\n[program:celery]\r\ncommand=/bin/bash -c \"source /home/web/www/.virtualenv/bin/activate && exec newrelic-admin run-program celery -A app.config worker -c 8 --loglevel=INFO -Q celery -n celery.%%h\"\r\nautostart=true\r\nautorestart=true\r\nstopsignal=KILL\r\nstdout_logfile=syslog\r\nstderr_logfile=syslog\r\nuser=ubuntu\r\ndirectory=\"...\"\r\nenvironment=\"...\"\r\nkillasgroup=true\r\nstopasgroup=true\r\n```\r\nAcross each the worker instance (specs: 16GB ram), we'll typically have a total task concurrency of ~90.  \r\n\r\nI managed to do an strace on the stuck process. \r\nSample:\r\n```\r\nsendto(26, \"\\316\", 1, MSG_NOSIGNAL, NULL, 0) = 1\r\ngettimeofday({1402061941, 289812}, NULL) = 0\r\ngettimeofday({1402061941, 289886}, NULL) = 0\r\nepoll_wait(3, {{EPOLLIN, {u32=25, u64=22420390710083609}}}, 1023, 1009) = 1\r\nselect(26, [25], NULL, NULL, {0, 0})    = 1 (in [25], left {0, 0})\r\nrecvfrom(25, \"\\1\\0\\2\\0\\0\\0)\\0<\\0<\\0012\\0\\0\\0\\0\\0\\34zP\\0\\10celeryev\\20\"..., 131072, 0, NULL, NULL) = 448\r\ngettimeofday({1402061941, 295328}, NULL) = 0\r\nepoll_ctl(3, EPOLL_CTL_DEL, 10, {EPOLLRDBAND|EPOLLWRNORM|EPOLLHUP|0x4f7020, {u32=0, u64=22420390710083584}}) = -1 ENOENT (No such file or directory)\r\nepoll_ctl(3, EPOLL_CTL_DEL, 14, {EPOLLRDBAND|EPOLLWRNORM|EPOLLHUP|0x4f7020, {u32=0, u64=22420390710083584}}) = -1 ENOENT (No such file or directory)\r\nepoll_ctl(3, EPOLL_CTL_DEL, 18, {EPOLLRDBAND|EPOLLWRNORM|EPOLLHUP|0x4f7020, {u32=0, u64=22420390710083584}}) = -1 ENOENT (No such file or directory)\r\nepoll_ctl(3, EPOLL_CTL_DEL, 6, {EPOLLRDBAND|EPOLLWRNORM|EPOLLHUP|0x4f7020, {u32=0, u64=22420390710083584}}) = -1 ENOENT (No such file or directory)\r\nselect(26, [25], NULL, NULL, {0, 0})    = 0 (Timeout)\r\ngettimeofday({1402061941, 295935}, NULL) = 0\r\n```\r\nMore detail here: https://gist.github.com/cameronmaske/db05f4676344ceece0e8\r\n\r\nWhen I manually killed the process ('kill -15 <pid>') the queue start up again.\r\n\r\nLet me know if there are anymore details that I can provide to help diagnose this! \r\n\r\n"},"comment":{"url":"https://api.github.com/repos/celery/celery/issues/comments/228217756","html_url":"https://github.com/celery/celery/issues/2080#issuecomment-228217756","issue_url":"https://api.github.com/repos/celery/celery/issues/2080","id":228217756,"user":{"login":"ask","id":1558,"avatar_url":"https://avatars.githubusercontent.com/u/1558?v=3","gravatar_id":"","url":"https://api.github.com/users/ask","html_url":"https://github.com/ask","followers_url":"https://api.github.com/users/ask/followers","following_url":"https://api.github.com/users/ask/following{/other_user}","gists_url":"https://api.github.com/users/ask/gists{/gist_id}","starred_url":"https://api.github.com/users/ask/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ask/subscriptions","organizations_url":"https://api.github.com/users/ask/orgs","repos_url":"https://api.github.com/users/ask/repos","events_url":"https://api.github.com/users/ask/events{/privacy}","received_events_url":"https://api.github.com/users/ask/received_events","type":"User","site_admin":false},"created_at":"2016-06-23T23:41:43Z","updated_at":"2016-06-23T23:41:43Z","body":"Closing this, as we don't have the resources to complete this task."}}
{"actor":{"display_login":"ask"}}