{"action":"created","issue":{"url":"https://api.github.com/repos/naparuba/shinken/issues/1875","repository_url":"https://api.github.com/repos/naparuba/shinken","labels_url":"https://api.github.com/repos/naparuba/shinken/issues/1875/labels{/name}","comments_url":"https://api.github.com/repos/naparuba/shinken/issues/1875/comments","events_url":"https://api.github.com/repos/naparuba/shinken/issues/1875/events","html_url":"https://github.com/naparuba/shinken/issues/1875","id":182471442,"number":1875,"title":"Actions never came back for the satellite 'poller-master'","user":{"login":"sanketplus","id":6016802,"avatar_url":"https://avatars.githubusercontent.com/u/6016802?v=3","gravatar_id":"","url":"https://api.github.com/users/sanketplus","html_url":"https://github.com/sanketplus","followers_url":"https://api.github.com/users/sanketplus/followers","following_url":"https://api.github.com/users/sanketplus/following{/other_user}","gists_url":"https://api.github.com/users/sanketplus/gists{/gist_id}","starred_url":"https://api.github.com/users/sanketplus/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sanketplus/subscriptions","organizations_url":"https://api.github.com/users/sanketplus/orgs","repos_url":"https://api.github.com/users/sanketplus/repos","events_url":"https://api.github.com/users/sanketplus/events{/privacy}","received_events_url":"https://api.github.com/users/sanketplus/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2016-10-12T09:00:11Z","updated_at":"2016-10-12T09:17:33Z","closed_at":null,"body":"Hey, \r\nI am running on latest 2.4.3 Things were running fine. checks were running and returning data. \r\n\r\nWhat happens is after some time, everything goes back into pending state. This has happened twice. I did a force recheck on one of the pending check  and I get following logs\r\n\r\n```\r\n==> arbiterd.log <==\r\n[1476262167] EXTERNAL COMMAND: [1476262157] SCHEDULE_FORCED_SVC_CHECK;<hostname>;CPU Idle;1476262157\r\n\r\n==> schedulerd.log <==\r\n[1476262189] WARNING: [Shinken] 774 actions never came back for the satellite 'poller-master'.I reenable them for polling\r\n\r\n==> pollerd.log <==\r\n[1476262266] ERROR: [Shinken] manage_returns exception:: <class 'shinken.http_client.HTTPException'>,Connection error to http://localhost:7768/ : Operation timed out after 3032 milliseconds with 0 bytes received \r\n[1476262266] INFO: [Shinken] [poller-master] Init connection with scheduler-master at http://localhost:7768/ (3s,120s)\r\n\r\n==> reactionnerd.log <==\r\n[1476262268] INFO: [Shinken] [reactionner-master] Init connection with scheduler-master at http://localhost:7768/ (3s,120s)\r\n\r\n==> pollerd.log <==\r\n[1476262269] WARNING: [Shinken] [poller-master] Scheduler scheduler-master is not initialized or has network problem: Connection error to http://localhost:7768/ : Operation timed out after 3032 milliseconds with 0 bytes received\r\n\r\n==> reactionnerd.log <==\r\n[1476262270] INFO: [Shinken] [reactionner-master] Connection OK with scheduler scheduler-master\r\n\r\n==> pollerd.log <==\r\n[1476262271] INFO: [Shinken] [poller-master] Init connection with scheduler-master at http://localhost:7768/ (3s,120s)\r\n[1476262271] INFO: [Shinken] [poller-master] Connection OK with scheduler scheduler-master\r\n\r\n==> schedulerd.log <==\r\n[1476262284] WARNING: [Shinken] 1226 actions never came back for the satellite 'poller-master'.I reenable them for polling\r\n[1476262377] WARNING: [Shinken] 1661 actions never came back for the satellite 'poller-master'.I reenable them for polling\r\n[1476262469] WARNING: [Shinken] 1731 actions never came back for the satellite 'poller-master'.I reenable them for polling\r\n```\r\n\r\nAll the daemons are running on same host. Installation method: pip\r\nOS:\r\nNAME=\"Amazon Linux AMI\"\r\nVERSION=\"2015.03\"\r\n\r\nThis relates to #1482 Please guide me further."},"comment":{"url":"https://api.github.com/repos/naparuba/shinken/issues/comments/253161776","html_url":"https://github.com/naparuba/shinken/issues/1875#issuecomment-253161776","issue_url":"https://api.github.com/repos/naparuba/shinken/issues/1875","id":253161776,"user":{"login":"sanketplus","id":6016802,"avatar_url":"https://avatars.githubusercontent.com/u/6016802?v=3","gravatar_id":"","url":"https://api.github.com/users/sanketplus","html_url":"https://github.com/sanketplus","followers_url":"https://api.github.com/users/sanketplus/followers","following_url":"https://api.github.com/users/sanketplus/following{/other_user}","gists_url":"https://api.github.com/users/sanketplus/gists{/gist_id}","starred_url":"https://api.github.com/users/sanketplus/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sanketplus/subscriptions","organizations_url":"https://api.github.com/users/sanketplus/orgs","repos_url":"https://api.github.com/users/sanketplus/repos","events_url":"https://api.github.com/users/sanketplus/events{/privacy}","received_events_url":"https://api.github.com/users/sanketplus/received_events","type":"User","site_admin":false},"created_at":"2016-10-12T09:17:32Z","updated_at":"2016-10-12T09:17:32Z","body":"This happens while everything seems to be running fine.\r\n```\r\nChecking status of scheduler\r\nscheduler RUNNING (pid 23237)\r\nChecking status of poller                                  [  OK  ]\r\npoller RUNNING (pid 23408)\r\nChecking status of reactionner                             [  OK  ]\r\nreactionner RUNNING (pid 23526)\r\nChecking status of broker                                  [  OK  ]\r\nbroker RUNNING (pid 23740)\r\nChecking status of receiver                                [  OK  ]\r\nreceiver RUNNING (pid 23988)\r\nChecking status of arbiter                                 [  OK  ]\r\narbiter RUNNING (pid 26513)\r\n[sanketp@monitor102 ~]$                                    [  OK  ]\r\n```\r\n\r\nI also enabled debugging in poller and the following keeps repeating in logs\r\n```\r\n[1476263694] DEBUG: [Shinken]  ======================== \r\n[1476263694] DEBUG: [Shinken] [0][scheduler-master][fork] Stats: Workers:0 (Queued:2996 TotalReturnWait:0)\r\n[1476263694] DEBUG: [Shinken] Wait ratio: 1.985084\r\n[1476263694] DEBUG: [Shinken] [poller-master] Trying to adjust worker number. Actual number : 1, min per module : 1, max per module : 1\r\n[1476263694] DEBUG: [Shinken] Ask actions to 0, got 4\r\n[1476263694] DEBUG: [Shinken] Loop turn\r\n[1476263694] DEBUG: [Shinken] Debug perf: ping [args:5.00679016113e-06] [aqu_lock:0.0][calling:1.90734863281e-06] [json:9.05990600586e-06] [global:1.59740447998e-05]\r\n[1476263694] DEBUG: [Shinken] HTTP: calling lock for get_broks\r\n[1476263694] DEBUG: [Shinken] Debug perf: get_broks [args:3.69548797607e-05] [aqu_lock:0.000128984451294][calling:4.00543212891e-05] [json:8.10623168945e-06] [global:0.000214099884033]\r\n[1476263695] DEBUG: [Shinken] Debug perf: ping [args:5.00679016113e-06] [aqu_lock:9.53674316406e-07][calling:1.90734863281e-06] [json:8.10623168945e-06] [global:1.59740447998e-05]\r\n[1476263695] DEBUG: [Shinken] Debug perf: get_external_commands [args:3.09944152832e-06] [aqu_lock:9.53674316406e-07][calling:1.59740447998e-05] [json:5.96046447754e-06] [global:2.59876251221e-05]\r\n[1476263696] DEBUG: [Shinken] Debug perf: ping [args:5.00679016113e-06] [aqu_lock:9.53674316406e-07][calling:2.14576721191e-06] [json:9.05990600586e-06] [global:1.71661376953e-05]\r\n[1476263696] DEBUG: [Shinken] HTTP: calling lock for get_broks\r\n[1476263696] DEBUG: [Shinken] Debug perf: get_broks [args:4.10079956055e-05] [aqu_lock:0.000244855880737][calling:6.103515625e-05] [json:1.00135803223e-05] [global:0.000356912612915]\r\n\r\n```"}}
{"actor":{"display_login":"sanketplus"}}