{"action":"created","issue":{"url":"https://api.github.com/repos/cantino/huginn/issues/1567","repository_url":"https://api.github.com/repos/cantino/huginn","labels_url":"https://api.github.com/repos/cantino/huginn/issues/1567/labels{/name}","comments_url":"https://api.github.com/repos/cantino/huginn/issues/1567/comments","events_url":"https://api.github.com/repos/cantino/huginn/issues/1567/events","html_url":"https://github.com/cantino/huginn/issues/1567","id":162856445,"number":1567,"title":"huginn background job stuck in running status no events output","user":{"login":"hughwi","id":5046683,"avatar_url":"https://avatars.githubusercontent.com/u/5046683?v=3","gravatar_id":"","url":"https://api.github.com/users/hughwi","html_url":"https://github.com/hughwi","followers_url":"https://api.github.com/users/hughwi/followers","following_url":"https://api.github.com/users/hughwi/following{/other_user}","gists_url":"https://api.github.com/users/hughwi/gists{/gist_id}","starred_url":"https://api.github.com/users/hughwi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hughwi/subscriptions","organizations_url":"https://api.github.com/users/hughwi/orgs","repos_url":"https://api.github.com/users/hughwi/repos","events_url":"https://api.github.com/users/hughwi/events{/privacy}","received_events_url":"https://api.github.com/users/hughwi/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":17,"created_at":"2016-06-29T07:14:51Z","updated_at":"2016-07-15T13:16:17Z","closed_at":null,"body":"I am having a similar issue to https://github.com/cantino/huginn/issues/864, except the job starts to run, and then gets stuck.\r\n\r\nThe system runs fine for a few weeks, and then will get stuck running a background job, which then causes a backlog in the queue. The only thing that fixes it is a restart of the application.\r\n\r\nI am running a Ubuntu Rackspace Cloud server on the default settings (as per the user guide), and resource usage looks fine (8vCPU and 8GB RAM (maximum usage is 5GB).\r\n\r\nWhat information do you need to help debug?"},"comment":{"url":"https://api.github.com/repos/cantino/huginn/issues/comments/232947901","html_url":"https://github.com/cantino/huginn/issues/1567#issuecomment-232947901","issue_url":"https://api.github.com/repos/cantino/huginn/issues/1567","id":232947901,"user":{"login":"dsander","id":20943,"avatar_url":"https://avatars.githubusercontent.com/u/20943?v=3","gravatar_id":"","url":"https://api.github.com/users/dsander","html_url":"https://github.com/dsander","followers_url":"https://api.github.com/users/dsander/followers","following_url":"https://api.github.com/users/dsander/following{/other_user}","gists_url":"https://api.github.com/users/dsander/gists{/gist_id}","starred_url":"https://api.github.com/users/dsander/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dsander/subscriptions","organizations_url":"https://api.github.com/users/dsander/orgs","repos_url":"https://api.github.com/users/dsander/repos","events_url":"https://api.github.com/users/dsander/events{/privacy}","received_events_url":"https://api.github.com/users/dsander/received_events","type":"User","site_admin":false},"created_at":"2016-07-15T13:16:16Z","updated_at":"2016-07-15T13:16:16Z","body":"To debug and fix the problem we either need a backtrace or a way to reproduce the issue. Does this always happen with the same WebsiteAgent or is it random?\r\n\r\nCan you check if the background worker is doing anything after it locked up?\r\n\r\n```\r\n# Get the PID\r\nps aux|grep threaded\r\n\r\nhuginn   31684  1.9 23.6 965028 240876 ?       Sl   Jul06 249:58 bin/threaded.rb\r\n\r\n# strace the main thread\r\nsudo strace -p 31684 -f\r\n\r\n# You should see output that looks like this:\r\nfutex(0x7f9b5a7ce794, FUTEX_WAIT_BITSET_PRIVATE, 6242125, {16932739, 665864126}, ffffffff) = -1 ETIMEDOUT (Connection timed out)\r\nfutex(0x7f9b5a7ce808, FUTEX_WAKE_PRIVATE, 1) = 0\r\nclock_gettime(CLOCK_MONOTONIC, {16932739, 666291235}) = 0\r\nclock_gettime(CLOCK_MONOTONIC, {16932739, 666401741}) = 0\r\nclock_gettime(CLOCK_MONOTONIC, {16932739, 666478470}) = 0\r\nfutex(0x7f9b5a7ce794, FUTEX_WAIT_BITSET_PRIVATE, 6242127, {16932739, 916478470}, ffffffff) = -1 ETIMEDOUT (Connection timed out)\r\nfutex(0x7f9b5a7ce808, FUTEX_WAKE_PRIVATE, 1) = 0\r\nclock_gettime(CLOCK_MONOTONIC, {16932739, 917375773}) = 0\r\nclock_gettime(CLOCK_MONOTONIC, {16932739, 917568375}) = 0\r\nclock_gettime(CLOCK_MONOTONIC, {16932739, 917736360}) = 0\r\nfutex(0x7f9b5a7ce794, FUTEX_WAIT_BITSET_PRIVATE, 6242129, {16932740, 167736360}, ffffffff) = -1 ETIMEDOUT (Connection timed out)\r\nfutex(0x7f9b5a7ce808, FUTEX_WAKE_PRIVATE, 1) = 0\r\n\r\n# strace all threads of the background worker and filter out clock_gettime and futex calls\r\nsudo strace -p 31684 -f -e 'trace=!clock_gettime,futex'\r\n\r\n# This should still output a bunch of system calls like `write`, `read` etc.\r\n```"}}
{"actor":{"display_login":"dsander"}}