{"action":"created","issue":{"url":"https://api.github.com/repos/celery/celery/issues/4840","repository_url":"https://api.github.com/repos/celery/celery","labels_url":"https://api.github.com/repos/celery/celery/issues/4840/labels{/name}","comments_url":"https://api.github.com/repos/celery/celery/issues/4840/comments","events_url":"https://api.github.com/repos/celery/celery/issues/4840/events","html_url":"https://github.com/celery/celery/issues/4840","id":334829428,"node_id":"MDU6SXNzdWUzMzQ4Mjk0Mjg=","number":4840,"title":"Celery lost / hanging messages without feedback (?)","user":{"login":"davidmir","id":4657051,"node_id":"MDQ6VXNlcjQ2NTcwNTE=","avatar_url":"https://avatars3.githubusercontent.com/u/4657051?v=4","gravatar_id":"","url":"https://api.github.com/users/davidmir","html_url":"https://github.com/davidmir","followers_url":"https://api.github.com/users/davidmir/followers","following_url":"https://api.github.com/users/davidmir/following{/other_user}","gists_url":"https://api.github.com/users/davidmir/gists{/gist_id}","starred_url":"https://api.github.com/users/davidmir/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/davidmir/subscriptions","organizations_url":"https://api.github.com/users/davidmir/orgs","repos_url":"https://api.github.com/users/davidmir/repos","events_url":"https://api.github.com/users/davidmir/events{/privacy}","received_events_url":"https://api.github.com/users/davidmir/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":9,"created_at":"2018-06-22T10:32:53Z","updated_at":"2018-06-27T15:38:39Z","closed_at":null,"author_association":"NONE","body":"Hi,\r\n\r\nI'm using Celery with RabbitMQ broker and I'm having the following problem: Messages are being published to the RabbitMQ queue, these messages are queued and consumed, but then the Celery process does not perform the task., however other times it processes the messages. My tasks can execute for a while (like 2 hours).\r\n\r\n**DEBUG**\r\n**celery logs:**\r\n I can't see nothing useful in the Celery logs (loglevel is in debug).\r\n\r\n**rabbitmq-logs:**\r\n\r\n> =WARNING REPORT==== 22-Jun-2018::10:17:07 ===\r\n> closing AMQP connection <0.2349.660> (172.17.0.1:38740 -> 172.17.0.2:5672, vhost: '/', user: 'guest):\r\n> client unexpectedly closed TCP connection\r\n\r\n**celery report:**\r\n\r\n> software -> celery:4.2.0 (windowlicker) kombu:4.2.1 py:2.7.9\r\n>             billiard:3.5.0.3 py-amqp:2.2.2\r\n> platform -> system:Linux arch:64bit, ELF imp:CPython\r\n> loader   -> celery.loaders.app.AppLoader\r\n> settings -> transport:amqp results:rpc:///\r\n\r\n**celery inspect status:**\r\n-> celery@6bd749f44e90: OK\r\n\r\n>     {\r\n>         \"broker\": {\r\n>             \"alternates\": [\r\n>                 \"amqp://guest:guest@rabbit1\",\r\n>                 \"amqp://guest:guest@rabbit2\"\r\n>             ],\r\n>             \"connect_timeout\": null,\r\n>             \"failover_strategy\": \"round-robin\",\r\n>             \"heartbeat\": 120.0,\r\n>             \"hostname\": \"rabbit1\",\r\n>             \"insist\": false,\r\n>             \"login_method\": \"AMQPLAIN\",\r\n>             \"port\": 5672,\r\n>             \"ssl\": false,\r\n>             \"transport\": \"amqp\",\r\n>             \"transport_options\": {},\r\n>             \"uri_prefix\": null,\r\n>             \"userid\": \"guest\",\r\n>             \"virtual_host\": \"/\"\r\n>         },\r\n>         \"clock\": \"48155359\",\r\n>         \"pid\": 10,\r\n>         \"pool\": {\r\n>             \"max-concurrency\": 2,\r\n>             \"max-tasks-per-child\": \"N/A\",\r\n>             \"processes\": [\r\n>                 15,\r\n>                 16\r\n>             ],\r\n>             \"put-guarded-by-semaphore\": false,\r\n>             \"timeouts\": [\r\n>                 0,\r\n>                 0\r\n>             ],\r\n>             \"writes\": {\r\n>                 \"all\": \"33.33%, 66.67%\",\r\n>                 \"avg\": \"44.44%\",\r\n>                 \"inqueues\": {\r\n>                     \"active\": 0,\r\n>                     \"total\": 2\r\n>                 },\r\n>                 \"raw\": \"3, 6\",\r\n>                 \"strategy\": null,\r\n>                 \"total\": 9\r\n>             }\r\n>         },\r\n>         \"prefetch_count\": 8,\r\n>         \"rusage\": {\r\n>             \"idrss\": 0,\r\n>             \"inblock\": 24,\r\n>             \"isrss\": 0,\r\n>             \"ixrss\": 0,\r\n>             \"majflt\": 0,\r\n>             \"maxrss\": 79736,\r\n>             \"minflt\": 32998,\r\n>             \"msgrcv\": 0,\r\n>             \"msgsnd\": 0,\r\n>             \"nivcsw\": 74723,\r\n>             \"nsignals\": 0,\r\n>             \"nswap\": 0,\r\n>             \"nvcsw\": 544907,\r\n>             \"oublock\": 752,\r\n>             \"stime\": 19.364,\r\n>             \"utime\": 206.084\r\n>         },\r\n>         \"total\": {\r\n>             \"my-task\": 9\r\n>         }\r\n>     }\r\n\r\nAnd sometime I see this:\r\n\r\n> celery@6bd749f44e90: OK\r\n>     {\r\n>         \"broker\": {\r\n>             \"alternates\": [\r\n>                 \"amqp://guest:guest@rabbit1:5672/\",\r\n>                 \"amqp://guest:guest@rabbit2:5672/\",\r\n>                 \"\"\r\n>             ],\r\n>             \"connect_timeout\": 4,\r\n>             \"failover_strategy\": \"round-robin\",\r\n>             \"heartbeat\": 120.0,\r\n>             \"hostname\": \"rabbit1\",\r\n>             \"insist\": false,\r\n>             \"login_method\": \"AMQPLAIN\",\r\n>             \"port\": 5672,\r\n>             \"ssl\": false,\r\n>             \"transport\": \"amqp\",\r\n>             \"transport_options\": {},\r\n>             \"uri_prefix\": null,\r\n>             \"userid\": \"guest\",\r\n>             \"virtual_host\": \"/\"\r\n>         },\r\n>         \"clock\": \"48176688\",\r\n>         \"pid\": 7,\r\n>         \"pool\": {\r\n>             \"max-concurrency\": 1,\r\n>             \"max-tasks-per-child\": \"N/A\",\r\n>             \"processes\": [\r\n>                 42\r\n>             ],\r\n>             \"put-guarded-by-semaphore\": false,\r\n>             \"timeouts\": [\r\n>                 0,\r\n>                 0\r\n>             ],\r\n>             \"writes\": {\r\n>                 \"all\": \"100.00%\",\r\n>                 \"avg\": \"100.00%\",\r\n>                 \"inqueues\": {\r\n>                     \"active\": 0,\r\n>                     \"total\": 1\r\n>                 },\r\n>                 \"raw\": \"1\",\r\n>                 \"strategy\": null,\r\n>                 \"total\": 1\r\n>             }\r\n>         },\r\n>         \"prefetch_count\": 4,\r\n>         \"rusage\": {\r\n>             \"idrss\": 0,\r\n>             \"inblock\": 0,\r\n>             \"isrss\": 0,\r\n>             \"ixrss\": 0,\r\n>             \"majflt\": 0,\r\n>             \"maxrss\": 313496,\r\n>             \"minflt\": 928413,\r\n>             \"msgrcv\": 0,\r\n>             \"msgsnd\": 0,\r\n>             \"nivcsw\": 341332,\r\n>             \"nsignals\": 0,\r\n>             \"nswap\": 0,\r\n>             \"nvcsw\": 8188755,\r\n>             \"oublock\": 29512,\r\n>             \"stime\": 111.844,\r\n>             \"utime\": 549.348\r\n>         },\r\n>         \"total\": {\r\n>             \"celery.backend_cleanup\": 1\r\n>         }\r\n>     }\r\n\r\n**strace:**\r\n\r\n> epoll_ctl(5, EPOLL_CTL_DEL, 16, 0x7fff97a45ae0) = -1 ENOENT (No such file or directory)\r\n> epoll_ctl(5, EPOLL_CTL_DEL, 21, 0x7fff97a45ae0) = -1 ENOENT (No such file or directory)\r\n> fcntl(26, F_GETFL)                      = 0x2 (flags O_RDWR)\r\n> fcntl(26, F_SETFL, O_RDWR|O_NONBLOCK)   = 0\r\n> recvfrom(26, 0x7fad09548924, 7, 0, NULL, NULL) = -1 EAGAIN (Resource temporarily unavailable)\r\n\r\nThis is strange. It says that have a total of 9 messages but Celery's processes are not processing these messages. CPU and Memory is stable.\r\n\r\n## Steps to reproduce\r\nPublish message with apply_asyc()\r\n\r\n## Expected behavior\r\nThe message is consumed from the RabbitMQ and the Celery worker process the message.\r\n\r\n## Actual behavior\r\nThe message is consumed from the RabbitMQ and the Celery worker does not process the message."},"comment":{"url":"https://api.github.com/repos/celery/celery/issues/comments/400721874","html_url":"https://github.com/celery/celery/issues/4840#issuecomment-400721874","issue_url":"https://api.github.com/repos/celery/celery/issues/4840","id":400721874,"node_id":"MDEyOklzc3VlQ29tbWVudDQwMDcyMTg3NA==","user":{"login":"mliudev","id":11674155,"node_id":"MDQ6VXNlcjExNjc0MTU1","avatar_url":"https://avatars0.githubusercontent.com/u/11674155?v=4","gravatar_id":"","url":"https://api.github.com/users/mliudev","html_url":"https://github.com/mliudev","followers_url":"https://api.github.com/users/mliudev/followers","following_url":"https://api.github.com/users/mliudev/following{/other_user}","gists_url":"https://api.github.com/users/mliudev/gists{/gist_id}","starred_url":"https://api.github.com/users/mliudev/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mliudev/subscriptions","organizations_url":"https://api.github.com/users/mliudev/orgs","repos_url":"https://api.github.com/users/mliudev/repos","events_url":"https://api.github.com/users/mliudev/events{/privacy}","received_events_url":"https://api.github.com/users/mliudev/received_events","type":"User","site_admin":false},"created_at":"2018-06-27T15:38:39Z","updated_at":"2018-06-27T15:38:39Z","author_association":"NONE","body":"I put a breakpoint in the task that's successful using `from celery.contrib import rdb; rdb.set_trace()` and walking the call stack. There are no errors popping up in the canvas code that calls `apply_async` after marking the previous task completed. However, I have noticed that the worker appears to die right after completing the task. I'm not sure if this is the reason why the subsequent tasks aren't executed. Flower and the worker both do not show that the next task has been received at all."}}
{"actor":{"display_login":"mliudev"}}