{"action":"created","issue":{"url":"https://api.github.com/repos/celery/celery/issues/4857","repository_url":"https://api.github.com/repos/celery/celery","labels_url":"https://api.github.com/repos/celery/celery/issues/4857/labels{/name}","comments_url":"https://api.github.com/repos/celery/celery/issues/4857/comments","events_url":"https://api.github.com/repos/celery/celery/issues/4857/events","html_url":"https://github.com/celery/celery/issues/4857","id":336229872,"node_id":"MDU6SXNzdWUzMzYyMjk4NzI=","number":4857,"title":"Redis results backend: apply_async().get() hangs forever after disconnection from redis-server","user":{"login":"amitlicht","id":10258630,"node_id":"MDQ6VXNlcjEwMjU4NjMw","avatar_url":"https://avatars0.githubusercontent.com/u/10258630?v=4","gravatar_id":"","url":"https://api.github.com/users/amitlicht","html_url":"https://github.com/amitlicht","followers_url":"https://api.github.com/users/amitlicht/followers","following_url":"https://api.github.com/users/amitlicht/following{/other_user}","gists_url":"https://api.github.com/users/amitlicht/gists{/gist_id}","starred_url":"https://api.github.com/users/amitlicht/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/amitlicht/subscriptions","organizations_url":"https://api.github.com/users/amitlicht/orgs","repos_url":"https://api.github.com/users/amitlicht/repos","events_url":"https://api.github.com/users/amitlicht/events{/privacy}","received_events_url":"https://api.github.com/users/amitlicht/received_events","type":"User","site_admin":false},"labels":[{"id":235025402,"node_id":"MDU6TGFiZWwyMzUwMjU0MDI=","url":"https://api.github.com/repos/celery/celery/labels/Component:%20Redis%20Results%20Backend","name":"Component: Redis Results Backend","color":"009800","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2018-06-27T13:27:29Z","updated_at":"2018-07-08T15:16:13Z","closed_at":null,"author_association":"NONE","body":"## Version info\r\n- celery: 4.2.0\r\n- kombu: 4.2.0\r\n- redis-py: 2.10.6\r\n- gevent: 1.2.2\r\n- redis server: 3.2.11 \r\n\r\n## Steps to reproduce\r\nCelery configuration: broker is rabbitmq, results backend is local redis (password protected). Both broker & client are gevented. \r\n1. start redis server\r\n2. start a celery worker\r\n3. call task.apply_async() once (tasks executes and output is returned)\r\n4. restart redis server\r\n5. call task.apply_async() again (caller hangs forever) \r\n\r\n## Expected behavior\r\n- During server downtime: apply_async().get() is expected to raise an error (`ConnectionError: Error while reading from socket: ('Connection closed by server.',)`)\r\n- After server is up again: apply_async().get() is expected to return to normal\r\n\r\n## Actual behavior\r\n- During server downtime: apply_async().get() sometimes raises an error and sometimes just hangs\r\n- After server is up again: apply_async().get() always hangs. Celery worker receives task request (over rabbitmq broker), executes it. Output is correctly stored in redis under celery-task-meta-TASKID. Caller never actually collects output and returns. \r\n\r\n## Additional information\r\n- Issue looks similar to https://github.com/celery/celery/issues/4556, however this one is not related to worker bootstrapping but rather to client hanging after disconnection. \r\n- Issue also reproduces (in fact, first occurred for us) when celery's redis client disconnects from the server due to network / proxy issues (in our case - middle-man HA proxy drops the connections). "},"comment":{"url":"https://api.github.com/repos/celery/celery/issues/comments/403294712","html_url":"https://github.com/celery/celery/issues/4857#issuecomment-403294712","issue_url":"https://api.github.com/repos/celery/celery/issues/4857","id":403294712,"node_id":"MDEyOklzc3VlQ29tbWVudDQwMzI5NDcxMg==","user":{"login":"amitlicht","id":10258630,"node_id":"MDQ6VXNlcjEwMjU4NjMw","avatar_url":"https://avatars0.githubusercontent.com/u/10258630?v=4","gravatar_id":"","url":"https://api.github.com/users/amitlicht","html_url":"https://github.com/amitlicht","followers_url":"https://api.github.com/users/amitlicht/followers","following_url":"https://api.github.com/users/amitlicht/following{/other_user}","gists_url":"https://api.github.com/users/amitlicht/gists{/gist_id}","starred_url":"https://api.github.com/users/amitlicht/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/amitlicht/subscriptions","organizations_url":"https://api.github.com/users/amitlicht/orgs","repos_url":"https://api.github.com/users/amitlicht/repos","events_url":"https://api.github.com/users/amitlicht/events{/privacy}","received_events_url":"https://api.github.com/users/amitlicht/received_events","type":"User","site_admin":false},"created_at":"2018-07-08T15:16:13Z","updated_at":"2018-07-08T15:16:13Z","author_association":"NONE","body":"@georgepsarakis \r\na. yes, I did, it had no effect on the issue. \r\nb. according to the documentation, the default value for this parameter is 120 seconds. Our process was stuck forever on `.get()`. \r\n\r\nI did try to circumvent the entire internal pub-sub usage within celery, by adding our own subscriber on `celery-task-meta-TASKID`, and having it call `on_state_change` on its own upon receiving messages. This bypass resolved the issue, that is, disconnections were caught and handled properly by redis-py (with practically no additional work from our client code). The internal celery consumer remains stuck, but the async_result is freed by our bypass consumer. \r\n\r\nJust throwing an idea out there - perhaps celery's usage of redis-py somehow disables redis-py disconnection handling? "}}
{"actor":{"display_login":"amitlicht"}}