{"action":"created","comment":{"url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/pulls/comments/72033597","id":72033597,"diff_hunk":"@@ -46,7 +46,11 @@ class Status:\n     CRITICAL = 2\n \n def check_status():\n-    output = subprocess.check_output(['monit', 'summary'])\n+    try:\n+        output = subprocess.check_output(['monit', 'summary'])\n+    except subprocess.CalledProcessError, e:","path":"clearwater-queue-manager.root/usr/share/clearwater/clearwater-queue-manager/scripts/check_node_health.py","position":7,"original_position":7,"commit_id":"5e0f1f14bb7c8b7855b0c639cb55fdf449bf9b20","original_commit_id":"5e0f1f14bb7c8b7855b0c639cb55fdf449bf9b20","user":{"login":"eleanor-merry","id":5425325,"avatar_url":"https://avatars.githubusercontent.com/u/5425325?v=3","gravatar_id":"","url":"https://api.github.com/users/eleanor-merry","html_url":"https://github.com/eleanor-merry","followers_url":"https://api.github.com/users/eleanor-merry/followers","following_url":"https://api.github.com/users/eleanor-merry/following{/other_user}","gists_url":"https://api.github.com/users/eleanor-merry/gists{/gist_id}","starred_url":"https://api.github.com/users/eleanor-merry/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/eleanor-merry/subscriptions","organizations_url":"https://api.github.com/users/eleanor-merry/orgs","repos_url":"https://api.github.com/users/eleanor-merry/repos","events_url":"https://api.github.com/users/eleanor-merry/events{/privacy}","received_events_url":"https://api.github.com/users/eleanor-merry/received_events","type":"User","site_admin":false},"body":"as e, not , e","created_at":"2016-07-25T09:27:58Z","updated_at":"2016-07-25T09:27:58Z","html_url":"https://github.com/Metaswitch/clearwater-etcd/pull/302#discussion_r72033597","pull_request_url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/pulls/302","_links":{"self":{"href":"https://api.github.com/repos/Metaswitch/clearwater-etcd/pulls/comments/72033597"},"html":{"href":"https://github.com/Metaswitch/clearwater-etcd/pull/302#discussion_r72033597"},"pull_request":{"href":"https://api.github.com/repos/Metaswitch/clearwater-etcd/pulls/302"}}},"pull_request":{"url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/pulls/302","id":78467694,"html_url":"https://github.com/Metaswitch/clearwater-etcd/pull/302","diff_url":"https://github.com/Metaswitch/clearwater-etcd/pull/302.diff","patch_url":"https://github.com/Metaswitch/clearwater-etcd/pull/302.patch","issue_url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/issues/302","number":302,"state":"open","locked":false,"title":"[Reviewer: Ellie] Catch socket connection error","user":{"login":"johadalin","id":14074163,"avatar_url":"https://avatars.githubusercontent.com/u/14074163?v=3","gravatar_id":"","url":"https://api.github.com/users/johadalin","html_url":"https://github.com/johadalin","followers_url":"https://api.github.com/users/johadalin/followers","following_url":"https://api.github.com/users/johadalin/following{/other_user}","gists_url":"https://api.github.com/users/johadalin/gists{/gist_id}","starred_url":"https://api.github.com/users/johadalin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/johadalin/subscriptions","organizations_url":"https://api.github.com/users/johadalin/orgs","repos_url":"https://api.github.com/users/johadalin/repos","events_url":"https://api.github.com/users/johadalin/events{/privacy}","received_events_url":"https://api.github.com/users/johadalin/received_events","type":"User","site_admin":false},"body":"The issue seen was that nodes would drop into the failed state in the queue-manager queue very quickly after starting the processing step. \r\nThe telling section of the queue-manager logs stated:\r\n```\r\n21-07-2016 15:50:52.233 UTC ERROR plugin_utils.py:70 (thread ApplyConfigPlugin): Command /usr/share/clearwater/clearwater-queue-manager/scripts/check_node_health.py failed with return code 1 and printed output 'Cannot create socket to [localhost]:2812 -- Connection refused\r\n```\r\n\r\nLooking at the check_node_health script, the subprocess.check_output call can raise a CalledProcessError exception that is only then caught once we have dropped out of the run loop, meaning that a single exception will kick us out of the retry for 450 seconds stage. \r\n\r\nCatching the error in place, and returning that call's status as 'CRITICAL' means that we will perform the run_loop correctly.\r\n\r\n\r\nThis was tested both as below, and also with the additions in the first commit to log to a temporary file, to see how often we hit the exception. \r\nRunning upload shared config to trigger the restart queue, the exception typically appeared on about 1 or two nodes in a 5 node deployment, and was only hit once in each cycle. I.e. we are not seeing multiple attempts and failures to open the socket to localhost:2812. \r\n\r\nThis socket is the Monit HTTP interface, which the monitrc file (https://github.com/Metaswitch/clearwater-monit/blob/master/monitrc) says the following on:\r\n\r\n```\r\n## Monit has an embedded HTTP interface which can be used to view status of \r\n## services monitored and manage services from a web interface. The HTTP \r\n## interface is also required if you want to issue Monit commands from the\r\n## command line, such as 'monit status' or 'monit restart service' The reason\r\n## for this is that the Monit client uses the HTTP interface to send these\r\n## commands to a running Monit daemon. See the Monit Wiki if you want to \r\n## enable SSL for the HTTP interface. \r\n#\r\nset httpd port 2812 and\r\n    use address localhost  # only accept connection from localhost\r\n    allow localhost        # allow localhost to connect to the server and\r\n    allow admin:monit      # require user 'admin' with password 'monit'\r\n```\r\n\r\nAs such, i think that this may be occuring as a side effect of moving up to the newer version of Monit, and it is taking slightly longer than before, or @eleanor-merry has suggested there may be some conflict with the mmonit.monit script. This needs some further investigation, but the fix in this PR should stop the exception pushing a node to the failed state.","created_at":"2016-07-22T12:14:17Z","updated_at":"2016-07-25T09:27:59Z","closed_at":null,"merged_at":null,"merge_commit_sha":"f1d77ee83b49f2b1e9ea6f9ec86d68fa60c22614","assignee":{"login":"eleanor-merry","id":5425325,"avatar_url":"https://avatars.githubusercontent.com/u/5425325?v=3","gravatar_id":"","url":"https://api.github.com/users/eleanor-merry","html_url":"https://github.com/eleanor-merry","followers_url":"https://api.github.com/users/eleanor-merry/followers","following_url":"https://api.github.com/users/eleanor-merry/following{/other_user}","gists_url":"https://api.github.com/users/eleanor-merry/gists{/gist_id}","starred_url":"https://api.github.com/users/eleanor-merry/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/eleanor-merry/subscriptions","organizations_url":"https://api.github.com/users/eleanor-merry/orgs","repos_url":"https://api.github.com/users/eleanor-merry/repos","events_url":"https://api.github.com/users/eleanor-merry/events{/privacy}","received_events_url":"https://api.github.com/users/eleanor-merry/received_events","type":"User","site_admin":false},"assignees":[{"login":"eleanor-merry","id":5425325,"avatar_url":"https://avatars.githubusercontent.com/u/5425325?v=3","gravatar_id":"","url":"https://api.github.com/users/eleanor-merry","html_url":"https://github.com/eleanor-merry","followers_url":"https://api.github.com/users/eleanor-merry/followers","following_url":"https://api.github.com/users/eleanor-merry/following{/other_user}","gists_url":"https://api.github.com/users/eleanor-merry/gists{/gist_id}","starred_url":"https://api.github.com/users/eleanor-merry/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/eleanor-merry/subscriptions","organizations_url":"https://api.github.com/users/eleanor-merry/orgs","repos_url":"https://api.github.com/users/eleanor-merry/repos","events_url":"https://api.github.com/users/eleanor-merry/events{/privacy}","received_events_url":"https://api.github.com/users/eleanor-merry/received_events","type":"User","site_admin":false}],"milestone":null,"commits_url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/pulls/302/commits","review_comments_url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/pulls/302/comments","review_comment_url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/pulls/comments{/number}","comments_url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/issues/302/comments","statuses_url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/statuses/5e0f1f14bb7c8b7855b0c639cb55fdf449bf9b20","head":{"label":"Metaswitch:catch_error","ref":"catch_error","sha":"5e0f1f14bb7c8b7855b0c639cb55fdf449bf9b20","user":{"login":"Metaswitch","id":453759,"avatar_url":"https://avatars.githubusercontent.com/u/453759?v=3","gravatar_id":"","url":"https://api.github.com/users/Metaswitch","html_url":"https://github.com/Metaswitch","followers_url":"https://api.github.com/users/Metaswitch/followers","following_url":"https://api.github.com/users/Metaswitch/following{/other_user}","gists_url":"https://api.github.com/users/Metaswitch/gists{/gist_id}","starred_url":"https://api.github.com/users/Metaswitch/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Metaswitch/subscriptions","organizations_url":"https://api.github.com/users/Metaswitch/orgs","repos_url":"https://api.github.com/users/Metaswitch/repos","events_url":"https://api.github.com/users/Metaswitch/events{/privacy}","received_events_url":"https://api.github.com/users/Metaswitch/received_events","type":"Organization","site_admin":false},"repo":{"id":34358654,"name":"clearwater-etcd","full_name":"Metaswitch/clearwater-etcd","owner":{"login":"Metaswitch","id":453759,"avatar_url":"https://avatars.githubusercontent.com/u/453759?v=3","gravatar_id":"","url":"https://api.github.com/users/Metaswitch","html_url":"https://github.com/Metaswitch","followers_url":"https://api.github.com/users/Metaswitch/followers","following_url":"https://api.github.com/users/Metaswitch/following{/other_user}","gists_url":"https://api.github.com/users/Metaswitch/gists{/gist_id}","starred_url":"https://api.github.com/users/Metaswitch/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Metaswitch/subscriptions","organizations_url":"https://api.github.com/users/Metaswitch/orgs","repos_url":"https://api.github.com/users/Metaswitch/repos","events_url":"https://api.github.com/users/Metaswitch/events{/privacy}","received_events_url":"https://api.github.com/users/Metaswitch/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/Metaswitch/clearwater-etcd","description":"Wrappers around etcd for ease of integration into Project Clearwater","fork":false,"url":"https://api.github.com/repos/Metaswitch/clearwater-etcd","forks_url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/forks","keys_url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/keys{/key_id}","collaborators_url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/teams","hooks_url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/hooks","issue_events_url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/issues/events{/number}","events_url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/events","assignees_url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/assignees{/user}","branches_url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/branches{/branch}","tags_url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/tags","blobs_url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/git/refs{/sha}","trees_url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/git/trees{/sha}","statuses_url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/statuses/{sha}","languages_url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/languages","stargazers_url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/stargazers","contributors_url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/contributors","subscribers_url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/subscribers","subscription_url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/subscription","commits_url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/commits{/sha}","git_commits_url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/git/commits{/sha}","comments_url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/comments{/number}","issue_comment_url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/issues/comments{/number}","contents_url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/contents/{+path}","compare_url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/compare/{base}...{head}","merges_url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/merges","archive_url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/downloads","issues_url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/issues{/number}","pulls_url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/pulls{/number}","milestones_url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/milestones{/number}","notifications_url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/labels{/name}","releases_url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/releases{/id}","deployments_url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/deployments","created_at":"2015-04-22T00:03:08Z","updated_at":"2016-01-11T14:37:25Z","pushed_at":"2016-07-22T12:14:19Z","git_url":"git://github.com/Metaswitch/clearwater-etcd.git","ssh_url":"git@github.com:Metaswitch/clearwater-etcd.git","clone_url":"https://github.com/Metaswitch/clearwater-etcd.git","svn_url":"https://github.com/Metaswitch/clearwater-etcd","homepage":null,"size":23575,"stargazers_count":0,"watchers_count":0,"language":"Python","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":2,"mirror_url":null,"open_issues_count":9,"forks":2,"open_issues":9,"watchers":0,"default_branch":"dev"}},"base":{"label":"Metaswitch:dev","ref":"dev","sha":"4d3d374f0f7fb32d58f9e181a23fa47b19f572d9","user":{"login":"Metaswitch","id":453759,"avatar_url":"https://avatars.githubusercontent.com/u/453759?v=3","gravatar_id":"","url":"https://api.github.com/users/Metaswitch","html_url":"https://github.com/Metaswitch","followers_url":"https://api.github.com/users/Metaswitch/followers","following_url":"https://api.github.com/users/Metaswitch/following{/other_user}","gists_url":"https://api.github.com/users/Metaswitch/gists{/gist_id}","starred_url":"https://api.github.com/users/Metaswitch/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Metaswitch/subscriptions","organizations_url":"https://api.github.com/users/Metaswitch/orgs","repos_url":"https://api.github.com/users/Metaswitch/repos","events_url":"https://api.github.com/users/Metaswitch/events{/privacy}","received_events_url":"https://api.github.com/users/Metaswitch/received_events","type":"Organization","site_admin":false},"repo":{"id":34358654,"name":"clearwater-etcd","full_name":"Metaswitch/clearwater-etcd","owner":{"login":"Metaswitch","id":453759,"avatar_url":"https://avatars.githubusercontent.com/u/453759?v=3","gravatar_id":"","url":"https://api.github.com/users/Metaswitch","html_url":"https://github.com/Metaswitch","followers_url":"https://api.github.com/users/Metaswitch/followers","following_url":"https://api.github.com/users/Metaswitch/following{/other_user}","gists_url":"https://api.github.com/users/Metaswitch/gists{/gist_id}","starred_url":"https://api.github.com/users/Metaswitch/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Metaswitch/subscriptions","organizations_url":"https://api.github.com/users/Metaswitch/orgs","repos_url":"https://api.github.com/users/Metaswitch/repos","events_url":"https://api.github.com/users/Metaswitch/events{/privacy}","received_events_url":"https://api.github.com/users/Metaswitch/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/Metaswitch/clearwater-etcd","description":"Wrappers around etcd for ease of integration into Project Clearwater","fork":false,"url":"https://api.github.com/repos/Metaswitch/clearwater-etcd","forks_url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/forks","keys_url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/keys{/key_id}","collaborators_url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/teams","hooks_url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/hooks","issue_events_url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/issues/events{/number}","events_url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/events","assignees_url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/assignees{/user}","branches_url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/branches{/branch}","tags_url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/tags","blobs_url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/git/refs{/sha}","trees_url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/git/trees{/sha}","statuses_url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/statuses/{sha}","languages_url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/languages","stargazers_url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/stargazers","contributors_url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/contributors","subscribers_url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/subscribers","subscription_url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/subscription","commits_url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/commits{/sha}","git_commits_url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/git/commits{/sha}","comments_url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/comments{/number}","issue_comment_url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/issues/comments{/number}","contents_url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/contents/{+path}","compare_url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/compare/{base}...{head}","merges_url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/merges","archive_url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/downloads","issues_url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/issues{/number}","pulls_url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/pulls{/number}","milestones_url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/milestones{/number}","notifications_url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/labels{/name}","releases_url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/releases{/id}","deployments_url":"https://api.github.com/repos/Metaswitch/clearwater-etcd/deployments","created_at":"2015-04-22T00:03:08Z","updated_at":"2016-01-11T14:37:25Z","pushed_at":"2016-07-22T12:14:19Z","git_url":"git://github.com/Metaswitch/clearwater-etcd.git","ssh_url":"git@github.com:Metaswitch/clearwater-etcd.git","clone_url":"https://github.com/Metaswitch/clearwater-etcd.git","svn_url":"https://github.com/Metaswitch/clearwater-etcd","homepage":null,"size":23575,"stargazers_count":0,"watchers_count":0,"language":"Python","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":2,"mirror_url":null,"open_issues_count":9,"forks":2,"open_issues":9,"watchers":0,"default_branch":"dev"}},"_links":{"self":{"href":"https://api.github.com/repos/Metaswitch/clearwater-etcd/pulls/302"},"html":{"href":"https://github.com/Metaswitch/clearwater-etcd/pull/302"},"issue":{"href":"https://api.github.com/repos/Metaswitch/clearwater-etcd/issues/302"},"comments":{"href":"https://api.github.com/repos/Metaswitch/clearwater-etcd/issues/302/comments"},"review_comments":{"href":"https://api.github.com/repos/Metaswitch/clearwater-etcd/pulls/302/comments"},"review_comment":{"href":"https://api.github.com/repos/Metaswitch/clearwater-etcd/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/Metaswitch/clearwater-etcd/pulls/302/commits"},"statuses":{"href":"https://api.github.com/repos/Metaswitch/clearwater-etcd/statuses/5e0f1f14bb7c8b7855b0c639cb55fdf449bf9b20"}}}}
{"actor":{"display_login":"eleanor-merry"}}