{"action":"created","issue":{"url":"https://api.github.com/repos/cdrx/faktory_worker_python/issues/10","repository_url":"https://api.github.com/repos/cdrx/faktory_worker_python","labels_url":"https://api.github.com/repos/cdrx/faktory_worker_python/issues/10/labels{/name}","comments_url":"https://api.github.com/repos/cdrx/faktory_worker_python/issues/10/comments","events_url":"https://api.github.com/repos/cdrx/faktory_worker_python/issues/10/events","html_url":"https://github.com/cdrx/faktory_worker_python/issues/10","id":294211023,"number":10,"title":"Worker falls into infinite loop receiving empty data from faktory","user":{"login":"valo","id":47639,"avatar_url":"https://avatars2.githubusercontent.com/u/47639?v=4","gravatar_id":"","url":"https://api.github.com/users/valo","html_url":"https://github.com/valo","followers_url":"https://api.github.com/users/valo/followers","following_url":"https://api.github.com/users/valo/following{/other_user}","gists_url":"https://api.github.com/users/valo/gists{/gist_id}","starred_url":"https://api.github.com/users/valo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/valo/subscriptions","organizations_url":"https://api.github.com/users/valo/orgs","repos_url":"https://api.github.com/users/valo/repos","events_url":"https://api.github.com/users/valo/events{/privacy}","received_events_url":"https://api.github.com/users/valo/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":9,"created_at":"2018-02-04T16:44:53Z","updated_at":"2018-02-06T17:23:02Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"I have a single threaded worker defined as:\r\n\r\n```python\r\nw = Worker(queues=['etherbi_decode'],\r\n           concurrency=1,\r\n           executor=ThreadPoolExecutor)\r\n```\r\n\r\nwhich works for a 2-3 hours and at some point falls into an infinite loop, not getting any new jobs and not sending heartbeats to the faktory server.\r\n\r\nLogs from the worker before it falls into infinite loop:\r\n\r\n```\r\nINFO:faktory.connection:Connecting to faktory-faktory:7419 (with password None)\r\nERROR:faktory.worker:Task failed: 50c676346b5a4307826de2245caf5593\r\nTraceback (most recent call last):\r\n  File \"/usr/local/lib/python3.6/concurrent/futures/thread.py\", line 56, in run\r\n    result = self.fn(*self.args, **self.kwargs)\r\n  File \"etherbi/decoder/decoder_worker.py\", line 225, in decode_bucket\r\n    _enqueue_block_decoded_task(task_number, 'calculate_burn_rate', 'etherbi_burn_rate_calculation')\r\n  File \"etherbi/decoder/decoder_worker.py\", line 187, in _enqueue_block_decoded_task\r\n    if faktory_client.queue(task, queue=queue, args=[task_number]):\r\n  File \"/app/src/faktory/faktory/client.py\", line 32, in queue\r\n    self.connect()\r\n  File \"/app/src/faktory/faktory/client.py\", line 21, in connect\r\n    self.is_connected = self.faktory.connect()\r\n  File \"/app/src/faktory/faktory/_proto.py\", line 64, in connect\r\n    self.socket.connect((self.host, self.port))\r\nsocket.timeout: timed out\r\nINFO:__main__:Transaction to 0xaf30d2a7e90d7dc361c8c4585e9bb7d2f6f15bc7 is recognized in block 3917149 on position 98.\r\nINFO:__main__:Transaction to 0xaf30d2a7e90d7dc361c8c4585e9bb7d2f6f15bc7 is recognized in block 3917154 on position 2.\r\n<SOME_WORK_RELATED_LOGS_AS_THE_ABOVE>\r\nINFO:faktory.connection:Connecting to faktory-faktory:7419 (with password None)\r\nINFO:faktory.connection:Disconnected\r\n```\r\n\r\nrunning `sudo strace -p <PID> -e trace=network -f -s 10000` against the process I get infinite stream of\r\n\r\n```\r\n[pid 16254] recvfrom(3, \"\", 4096, 0, NULL, NULL) = 0\r\n[pid 16254] recvfrom(3, \"\", 4096, 0, NULL, NULL) = 0\r\n[pid 16254] recvfrom(3, \"\", 4096, 0, NULL, NULL) = 0\r\n[pid 16254] recvfrom(3, \"\", 4096, 0, NULL, NULL) = 0\r\n[pid 16254] recvfrom(3, \"\", 4096, 0, NULL, NULL) = 0\r\n[pid 16254] recvfrom(3, \"\", 4096, 0, NULL, NULL) = 0\r\n[pid 16254] recvfrom(3, \"\", 4096, 0, NULL, NULL) = 0\r\n[pid 16254] recvfrom(3, \"\", 4096, 0, NULL, NULL) = 0\r\n```\r\n\r\nWe use python and elixir clients against the same faktory server and so far this behavior is observed only with the python workers.\r\n\r\nWorker version: c5cb89b86aa0e448bb010d237a35a0f621db3577\r\nServer version: 0.7.0"},"comment":{"url":"https://api.github.com/repos/cdrx/faktory_worker_python/issues/comments/363498116","html_url":"https://github.com/cdrx/faktory_worker_python/issues/10#issuecomment-363498116","issue_url":"https://api.github.com/repos/cdrx/faktory_worker_python/issues/10","id":363498116,"user":{"login":"valo","id":47639,"avatar_url":"https://avatars2.githubusercontent.com/u/47639?v=4","gravatar_id":"","url":"https://api.github.com/users/valo","html_url":"https://github.com/valo","followers_url":"https://api.github.com/users/valo/followers","following_url":"https://api.github.com/users/valo/following{/other_user}","gists_url":"https://api.github.com/users/valo/gists{/gist_id}","starred_url":"https://api.github.com/users/valo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/valo/subscriptions","organizations_url":"https://api.github.com/users/valo/orgs","repos_url":"https://api.github.com/users/valo/repos","events_url":"https://api.github.com/users/valo/events{/privacy}","received_events_url":"https://api.github.com/users/valo/received_events","type":"User","site_admin":false},"created_at":"2018-02-06T17:23:02Z","updated_at":"2018-02-06T17:23:02Z","author_association":"CONTRIBUTOR","body":"For sure we are not seeing this behavior with `faktory` 0.6.1. We've been rolling with it for the past 2 days, while everything else is the same and not problems so far."}}
{"actor":{"display_login":"valo"}}