{"action":"created","comment":{"url":"https://api.github.com/repos/crowbar/crowbar-openstack/pulls/comments/173435878","pull_request_review_id":102625259,"id":173435878,"diff_hunk":"@@ -0,0 +1,36 @@\n+#!/bin/sh\n+\n+# get the number of running nodes of rabbitmq in the current cluster\n+function running_nodes()\n+{\n+  local running_nodes=$(rabbitmqctl cluster_status 2>/dev/null | grep running_nodes | cut -d \",\" -f2- | cut -d \"}\" -f1 | tr -d \"[\" | tr -d \"]\" | tr -d \"'\" | tr \",\" \"\\n\" | wc -l)\n+  echo $running_nodes","path":"chef/cookbooks/rabbitmq/templates/default/rabbitmq-port-blocker.erb","position":7,"original_position":7,"commit_id":"17e6e11d1c14be739dc13537ee200d1cd5195e6d","original_commit_id":"17e6e11d1c14be739dc13537ee200d1cd5195e6d","user":{"login":"vuntz","id":1758783,"avatar_url":"https://avatars3.githubusercontent.com/u/1758783?v=4","gravatar_id":"","url":"https://api.github.com/users/vuntz","html_url":"https://github.com/vuntz","followers_url":"https://api.github.com/users/vuntz/followers","following_url":"https://api.github.com/users/vuntz/following{/other_user}","gists_url":"https://api.github.com/users/vuntz/gists{/gist_id}","starred_url":"https://api.github.com/users/vuntz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vuntz/subscriptions","organizations_url":"https://api.github.com/users/vuntz/orgs","repos_url":"https://api.github.com/users/vuntz/repos","events_url":"https://api.github.com/users/vuntz/events{/privacy}","received_events_url":"https://api.github.com/users/vuntz/received_events","type":"User","site_admin":false},"body":"Btw, I'm assuming the command will work fine even when a cluster has 5 nodes.","created_at":"2018-03-09T12:19:46Z","updated_at":"2018-03-09T12:22:55Z","html_url":"https://github.com/crowbar/crowbar-openstack/pull/1554#discussion_r173435878","pull_request_url":"https://api.github.com/repos/crowbar/crowbar-openstack/pulls/1554","author_association":"OWNER","_links":{"self":{"href":"https://api.github.com/repos/crowbar/crowbar-openstack/pulls/comments/173435878"},"html":{"href":"https://github.com/crowbar/crowbar-openstack/pull/1554#discussion_r173435878"},"pull_request":{"href":"https://api.github.com/repos/crowbar/crowbar-openstack/pulls/1554"}},"in_reply_to_id":173435211},"pull_request":{"url":"https://api.github.com/repos/crowbar/crowbar-openstack/pulls/1554","id":168254194,"html_url":"https://github.com/crowbar/crowbar-openstack/pull/1554","diff_url":"https://github.com/crowbar/crowbar-openstack/pull/1554.diff","patch_url":"https://github.com/crowbar/crowbar-openstack/pull/1554.patch","issue_url":"https://api.github.com/repos/crowbar/crowbar-openstack/issues/1554","number":1554,"state":"open","locked":false,"title":"rabbitmq: block client port on startup","user":{"login":"ilausuch","id":11134265,"avatar_url":"https://avatars2.githubusercontent.com/u/11134265?v=4","gravatar_id":"","url":"https://api.github.com/users/ilausuch","html_url":"https://github.com/ilausuch","followers_url":"https://api.github.com/users/ilausuch/followers","following_url":"https://api.github.com/users/ilausuch/following{/other_user}","gists_url":"https://api.github.com/users/ilausuch/gists{/gist_id}","starred_url":"https://api.github.com/users/ilausuch/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ilausuch/subscriptions","organizations_url":"https://api.github.com/users/ilausuch/orgs","repos_url":"https://api.github.com/users/ilausuch/repos","events_url":"https://api.github.com/users/ilausuch/events{/privacy}","received_events_url":"https://api.github.com/users/ilausuch/received_events","type":"User","site_admin":false},"body":"Sometimes when a cluster of rabbitmq nodes are restarted the first node will have an excessive number of master queues creation and an excessive number of concurrent connections to manage, this generates timeout to the clients. These clients try to reconnect generating more traffic over this overloaded node. This could happen too in case the number of nodes is below to a critical level (In the script is fixed to the half of total nodes in cluster)\r\n\r\nThis scripts solves the problem blocking all rabbitmq client connections (via iptables) if the aren't enough nodes. When the problem has gone unblocks the ports. \r\n\r\nTo achieve that the script will take advantage from the pacemaker notifications. When the pacemaker detects that trabbitmq has failed on a node will send a monitor notification to this node. In this point the condition to block the rabbitmq client port has to be evaluated, and block or not the ports in all cluster nodes. Because the other nodes won't be notified about this event, is the problematic node the responsible to notify other nodes. When the rabbitmq of a node has recovered pacemaker will send a notification to this node, if the global condition is normal will unblock the port, and all other nodes will be notified to do the same.\r\n\r\n\r\nThe script is divided in two parts: On one hand, the alert handler that manages the alerts discarding non interesting alerts, launches to the blocker script in this node and other cluster nodes (via SSH). And in the other hand, the blocker script than checks the condition of running nodes and blocks or unblocks the rabbitmq client port.","created_at":"2018-02-09T14:33:16Z","updated_at":"2018-03-09T12:22:55Z","closed_at":null,"merged_at":null,"merge_commit_sha":"75edfee12cd53647180b5f5cf39797e124934613","assignee":null,"assignees":[],"requested_reviewers":[{"login":"AbelNavarro","id":1333236,"avatar_url":"https://avatars1.githubusercontent.com/u/1333236?v=4","gravatar_id":"","url":"https://api.github.com/users/AbelNavarro","html_url":"https://github.com/AbelNavarro","followers_url":"https://api.github.com/users/AbelNavarro/followers","following_url":"https://api.github.com/users/AbelNavarro/following{/other_user}","gists_url":"https://api.github.com/users/AbelNavarro/gists{/gist_id}","starred_url":"https://api.github.com/users/AbelNavarro/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AbelNavarro/subscriptions","organizations_url":"https://api.github.com/users/AbelNavarro/orgs","repos_url":"https://api.github.com/users/AbelNavarro/repos","events_url":"https://api.github.com/users/AbelNavarro/events{/privacy}","received_events_url":"https://api.github.com/users/AbelNavarro/received_events","type":"User","site_admin":false},{"login":"rsalevsky","id":4128547,"avatar_url":"https://avatars1.githubusercontent.com/u/4128547?v=4","gravatar_id":"","url":"https://api.github.com/users/rsalevsky","html_url":"https://github.com/rsalevsky","followers_url":"https://api.github.com/users/rsalevsky/followers","following_url":"https://api.github.com/users/rsalevsky/following{/other_user}","gists_url":"https://api.github.com/users/rsalevsky/gists{/gist_id}","starred_url":"https://api.github.com/users/rsalevsky/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rsalevsky/subscriptions","organizations_url":"https://api.github.com/users/rsalevsky/orgs","repos_url":"https://api.github.com/users/rsalevsky/repos","events_url":"https://api.github.com/users/rsalevsky/events{/privacy}","received_events_url":"https://api.github.com/users/rsalevsky/received_events","type":"User","site_admin":false},{"login":"bbobrov","id":33933950,"avatar_url":"https://avatars1.githubusercontent.com/u/33933950?v=4","gravatar_id":"","url":"https://api.github.com/users/bbobrov","html_url":"https://github.com/bbobrov","followers_url":"https://api.github.com/users/bbobrov/followers","following_url":"https://api.github.com/users/bbobrov/following{/other_user}","gists_url":"https://api.github.com/users/bbobrov/gists{/gist_id}","starred_url":"https://api.github.com/users/bbobrov/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bbobrov/subscriptions","organizations_url":"https://api.github.com/users/bbobrov/orgs","repos_url":"https://api.github.com/users/bbobrov/repos","events_url":"https://api.github.com/users/bbobrov/events{/privacy}","received_events_url":"https://api.github.com/users/bbobrov/received_events","type":"User","site_admin":false}],"requested_teams":[],"labels":[{"id":711258318,"url":"https://api.github.com/repos/crowbar/crowbar-openstack/labels/needs%20backport%20to%20SOC7","name":"needs backport to SOC7","color":"0052cc","default":false}],"milestone":null,"commits_url":"https://api.github.com/repos/crowbar/crowbar-openstack/pulls/1554/commits","review_comments_url":"https://api.github.com/repos/crowbar/crowbar-openstack/pulls/1554/comments","review_comment_url":"https://api.github.com/repos/crowbar/crowbar-openstack/pulls/comments{/number}","comments_url":"https://api.github.com/repos/crowbar/crowbar-openstack/issues/1554/comments","statuses_url":"https://api.github.com/repos/crowbar/crowbar-openstack/statuses/17e6e11d1c14be739dc13537ee200d1cd5195e6d","head":{"label":"ilausuch:rabbitmq_block_client_port_on_startup","ref":"rabbitmq_block_client_port_on_startup","sha":"17e6e11d1c14be739dc13537ee200d1cd5195e6d","user":{"login":"ilausuch","id":11134265,"avatar_url":"https://avatars2.githubusercontent.com/u/11134265?v=4","gravatar_id":"","url":"https://api.github.com/users/ilausuch","html_url":"https://github.com/ilausuch","followers_url":"https://api.github.com/users/ilausuch/followers","following_url":"https://api.github.com/users/ilausuch/following{/other_user}","gists_url":"https://api.github.com/users/ilausuch/gists{/gist_id}","starred_url":"https://api.github.com/users/ilausuch/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ilausuch/subscriptions","organizations_url":"https://api.github.com/users/ilausuch/orgs","repos_url":"https://api.github.com/users/ilausuch/repos","events_url":"https://api.github.com/users/ilausuch/events{/privacy}","received_events_url":"https://api.github.com/users/ilausuch/received_events","type":"User","site_admin":false},"repo":{"id":107387251,"name":"crowbar-openstack","full_name":"ilausuch/crowbar-openstack","owner":{"login":"ilausuch","id":11134265,"avatar_url":"https://avatars2.githubusercontent.com/u/11134265?v=4","gravatar_id":"","url":"https://api.github.com/users/ilausuch","html_url":"https://github.com/ilausuch","followers_url":"https://api.github.com/users/ilausuch/followers","following_url":"https://api.github.com/users/ilausuch/following{/other_user}","gists_url":"https://api.github.com/users/ilausuch/gists{/gist_id}","starred_url":"https://api.github.com/users/ilausuch/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ilausuch/subscriptions","organizations_url":"https://api.github.com/users/ilausuch/orgs","repos_url":"https://api.github.com/users/ilausuch/repos","events_url":"https://api.github.com/users/ilausuch/events{/privacy}","received_events_url":"https://api.github.com/users/ilausuch/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/ilausuch/crowbar-openstack","description":"Openstack deployment for Crowbar","fork":true,"url":"https://api.github.com/repos/ilausuch/crowbar-openstack","forks_url":"https://api.github.com/repos/ilausuch/crowbar-openstack/forks","keys_url":"https://api.github.com/repos/ilausuch/crowbar-openstack/keys{/key_id}","collaborators_url":"https://api.github.com/repos/ilausuch/crowbar-openstack/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/ilausuch/crowbar-openstack/teams","hooks_url":"https://api.github.com/repos/ilausuch/crowbar-openstack/hooks","issue_events_url":"https://api.github.com/repos/ilausuch/crowbar-openstack/issues/events{/number}","events_url":"https://api.github.com/repos/ilausuch/crowbar-openstack/events","assignees_url":"https://api.github.com/repos/ilausuch/crowbar-openstack/assignees{/user}","branches_url":"https://api.github.com/repos/ilausuch/crowbar-openstack/branches{/branch}","tags_url":"https://api.github.com/repos/ilausuch/crowbar-openstack/tags","blobs_url":"https://api.github.com/repos/ilausuch/crowbar-openstack/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/ilausuch/crowbar-openstack/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/ilausuch/crowbar-openstack/git/refs{/sha}","trees_url":"https://api.github.com/repos/ilausuch/crowbar-openstack/git/trees{/sha}","statuses_url":"https://api.github.com/repos/ilausuch/crowbar-openstack/statuses/{sha}","languages_url":"https://api.github.com/repos/ilausuch/crowbar-openstack/languages","stargazers_url":"https://api.github.com/repos/ilausuch/crowbar-openstack/stargazers","contributors_url":"https://api.github.com/repos/ilausuch/crowbar-openstack/contributors","subscribers_url":"https://api.github.com/repos/ilausuch/crowbar-openstack/subscribers","subscription_url":"https://api.github.com/repos/ilausuch/crowbar-openstack/subscription","commits_url":"https://api.github.com/repos/ilausuch/crowbar-openstack/commits{/sha}","git_commits_url":"https://api.github.com/repos/ilausuch/crowbar-openstack/git/commits{/sha}","comments_url":"https://api.github.com/repos/ilausuch/crowbar-openstack/comments{/number}","issue_comment_url":"https://api.github.com/repos/ilausuch/crowbar-openstack/issues/comments{/number}","contents_url":"https://api.github.com/repos/ilausuch/crowbar-openstack/contents/{+path}","compare_url":"https://api.github.com/repos/ilausuch/crowbar-openstack/compare/{base}...{head}","merges_url":"https://api.github.com/repos/ilausuch/crowbar-openstack/merges","archive_url":"https://api.github.com/repos/ilausuch/crowbar-openstack/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/ilausuch/crowbar-openstack/downloads","issues_url":"https://api.github.com/repos/ilausuch/crowbar-openstack/issues{/number}","pulls_url":"https://api.github.com/repos/ilausuch/crowbar-openstack/pulls{/number}","milestones_url":"https://api.github.com/repos/ilausuch/crowbar-openstack/milestones{/number}","notifications_url":"https://api.github.com/repos/ilausuch/crowbar-openstack/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/ilausuch/crowbar-openstack/labels{/name}","releases_url":"https://api.github.com/repos/ilausuch/crowbar-openstack/releases{/id}","deployments_url":"https://api.github.com/repos/ilausuch/crowbar-openstack/deployments","created_at":"2017-10-18T09:33:23Z","updated_at":"2017-10-18T09:33:27Z","pushed_at":"2018-03-08T08:56:11Z","git_url":"git://github.com/ilausuch/crowbar-openstack.git","ssh_url":"git@github.com:ilausuch/crowbar-openstack.git","clone_url":"https://github.com/ilausuch/crowbar-openstack.git","svn_url":"https://github.com/ilausuch/crowbar-openstack","homepage":null,"size":20803,"stargazers_count":0,"watchers_count":0,"language":"Ruby","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"archived":false,"open_issues_count":0,"license":null,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"crowbar:master","ref":"master","sha":"d2273ddac065e6f1fee7978be6ccd66a90449130","user":{"login":"crowbar","id":2910205,"avatar_url":"https://avatars2.githubusercontent.com/u/2910205?v=4","gravatar_id":"","url":"https://api.github.com/users/crowbar","html_url":"https://github.com/crowbar","followers_url":"https://api.github.com/users/crowbar/followers","following_url":"https://api.github.com/users/crowbar/following{/other_user}","gists_url":"https://api.github.com/users/crowbar/gists{/gist_id}","starred_url":"https://api.github.com/users/crowbar/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/crowbar/subscriptions","organizations_url":"https://api.github.com/users/crowbar/orgs","repos_url":"https://api.github.com/users/crowbar/repos","events_url":"https://api.github.com/users/crowbar/events{/privacy}","received_events_url":"https://api.github.com/users/crowbar/received_events","type":"Organization","site_admin":false},"repo":{"id":39006903,"name":"crowbar-openstack","full_name":"crowbar/crowbar-openstack","owner":{"login":"crowbar","id":2910205,"avatar_url":"https://avatars2.githubusercontent.com/u/2910205?v=4","gravatar_id":"","url":"https://api.github.com/users/crowbar","html_url":"https://github.com/crowbar","followers_url":"https://api.github.com/users/crowbar/followers","following_url":"https://api.github.com/users/crowbar/following{/other_user}","gists_url":"https://api.github.com/users/crowbar/gists{/gist_id}","starred_url":"https://api.github.com/users/crowbar/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/crowbar/subscriptions","organizations_url":"https://api.github.com/users/crowbar/orgs","repos_url":"https://api.github.com/users/crowbar/repos","events_url":"https://api.github.com/users/crowbar/events{/privacy}","received_events_url":"https://api.github.com/users/crowbar/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/crowbar/crowbar-openstack","description":"Openstack deployment for Crowbar","fork":false,"url":"https://api.github.com/repos/crowbar/crowbar-openstack","forks_url":"https://api.github.com/repos/crowbar/crowbar-openstack/forks","keys_url":"https://api.github.com/repos/crowbar/crowbar-openstack/keys{/key_id}","collaborators_url":"https://api.github.com/repos/crowbar/crowbar-openstack/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/crowbar/crowbar-openstack/teams","hooks_url":"https://api.github.com/repos/crowbar/crowbar-openstack/hooks","issue_events_url":"https://api.github.com/repos/crowbar/crowbar-openstack/issues/events{/number}","events_url":"https://api.github.com/repos/crowbar/crowbar-openstack/events","assignees_url":"https://api.github.com/repos/crowbar/crowbar-openstack/assignees{/user}","branches_url":"https://api.github.com/repos/crowbar/crowbar-openstack/branches{/branch}","tags_url":"https://api.github.com/repos/crowbar/crowbar-openstack/tags","blobs_url":"https://api.github.com/repos/crowbar/crowbar-openstack/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/crowbar/crowbar-openstack/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/crowbar/crowbar-openstack/git/refs{/sha}","trees_url":"https://api.github.com/repos/crowbar/crowbar-openstack/git/trees{/sha}","statuses_url":"https://api.github.com/repos/crowbar/crowbar-openstack/statuses/{sha}","languages_url":"https://api.github.com/repos/crowbar/crowbar-openstack/languages","stargazers_url":"https://api.github.com/repos/crowbar/crowbar-openstack/stargazers","contributors_url":"https://api.github.com/repos/crowbar/crowbar-openstack/contributors","subscribers_url":"https://api.github.com/repos/crowbar/crowbar-openstack/subscribers","subscription_url":"https://api.github.com/repos/crowbar/crowbar-openstack/subscription","commits_url":"https://api.github.com/repos/crowbar/crowbar-openstack/commits{/sha}","git_commits_url":"https://api.github.com/repos/crowbar/crowbar-openstack/git/commits{/sha}","comments_url":"https://api.github.com/repos/crowbar/crowbar-openstack/comments{/number}","issue_comment_url":"https://api.github.com/repos/crowbar/crowbar-openstack/issues/comments{/number}","contents_url":"https://api.github.com/repos/crowbar/crowbar-openstack/contents/{+path}","compare_url":"https://api.github.com/repos/crowbar/crowbar-openstack/compare/{base}...{head}","merges_url":"https://api.github.com/repos/crowbar/crowbar-openstack/merges","archive_url":"https://api.github.com/repos/crowbar/crowbar-openstack/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/crowbar/crowbar-openstack/downloads","issues_url":"https://api.github.com/repos/crowbar/crowbar-openstack/issues{/number}","pulls_url":"https://api.github.com/repos/crowbar/crowbar-openstack/pulls{/number}","milestones_url":"https://api.github.com/repos/crowbar/crowbar-openstack/milestones{/number}","notifications_url":"https://api.github.com/repos/crowbar/crowbar-openstack/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/crowbar/crowbar-openstack/labels{/name}","releases_url":"https://api.github.com/repos/crowbar/crowbar-openstack/releases{/id}","deployments_url":"https://api.github.com/repos/crowbar/crowbar-openstack/deployments","created_at":"2015-07-13T10:55:33Z","updated_at":"2018-03-09T05:07:31Z","pushed_at":"2018-03-09T09:12:12Z","git_url":"git://github.com/crowbar/crowbar-openstack.git","ssh_url":"git@github.com:crowbar/crowbar-openstack.git","clone_url":"https://github.com/crowbar/crowbar-openstack.git","svn_url":"https://github.com/crowbar/crowbar-openstack","homepage":null,"size":20829,"stargazers_count":15,"watchers_count":15,"language":"Ruby","has_issues":false,"has_projects":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":78,"mirror_url":null,"archived":false,"open_issues_count":74,"license":null,"forks":78,"open_issues":74,"watchers":15,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/crowbar/crowbar-openstack/pulls/1554"},"html":{"href":"https://github.com/crowbar/crowbar-openstack/pull/1554"},"issue":{"href":"https://api.github.com/repos/crowbar/crowbar-openstack/issues/1554"},"comments":{"href":"https://api.github.com/repos/crowbar/crowbar-openstack/issues/1554/comments"},"review_comments":{"href":"https://api.github.com/repos/crowbar/crowbar-openstack/pulls/1554/comments"},"review_comment":{"href":"https://api.github.com/repos/crowbar/crowbar-openstack/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/crowbar/crowbar-openstack/pulls/1554/commits"},"statuses":{"href":"https://api.github.com/repos/crowbar/crowbar-openstack/statuses/17e6e11d1c14be739dc13537ee200d1cd5195e6d"}},"author_association":"CONTRIBUTOR"}}
{"actor":{"display_login":"vuntz"}}