{"action":"created","issue":{"url":"https://api.github.com/repos/openvstorage/framework/issues/860","repository_url":"https://api.github.com/repos/openvstorage/framework","labels_url":"https://api.github.com/repos/openvstorage/framework/issues/860/labels{/name}","comments_url":"https://api.github.com/repos/openvstorage/framework/issues/860/comments","events_url":"https://api.github.com/repos/openvstorage/framework/issues/860/events","html_url":"https://github.com/openvstorage/framework/issues/860","id":173956184,"number":860,"title":"Rabbitmq partitions problems","user":{"login":"jeroenmaelbrancke","id":2900612,"avatar_url":"https://avatars.githubusercontent.com/u/2900612?v=3","gravatar_id":"","url":"https://api.github.com/users/jeroenmaelbrancke","html_url":"https://github.com/jeroenmaelbrancke","followers_url":"https://api.github.com/users/jeroenmaelbrancke/followers","following_url":"https://api.github.com/users/jeroenmaelbrancke/following{/other_user}","gists_url":"https://api.github.com/users/jeroenmaelbrancke/gists{/gist_id}","starred_url":"https://api.github.com/users/jeroenmaelbrancke/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jeroenmaelbrancke/subscriptions","organizations_url":"https://api.github.com/users/jeroenmaelbrancke/orgs","repos_url":"https://api.github.com/users/jeroenmaelbrancke/repos","events_url":"https://api.github.com/users/jeroenmaelbrancke/events{/privacy}","received_events_url":"https://api.github.com/users/jeroenmaelbrancke/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/openvstorage/framework/labels/SRP","name":"SRP","color":"5319e7"}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2016-08-30T08:14:02Z","updated_at":"2016-08-30T14:40:13Z","closed_at":null,"body":"On the OVH setup we had a partition issues on rabbitmq.\r\nAll the ovs-workers got stuck and didn't process any task anymore.\r\non the webgui you were not able to see the backend, vpools, vdisk,....\r\n\r\nOutput form the rabbitmq cluster\r\n```\r\nroot@ctrl-stras-03:~# rabbitmqctl cluster_status\r\nCluster status of node 'rabbit@ctrl-stras-03' ...\r\n[{nodes,[{disc,['rabbit@ctrl-grav-02','rabbit@ctrl-roub-01',\r\n                'rabbit@ctrl-stras-03']}]},\r\n {running_nodes,['rabbit@ctrl-roub-01','rabbit@ctrl-grav-02',\r\n                 'rabbit@ctrl-stras-03']},\r\n {partitions,[{'rabbit@ctrl-roub-01',['rabbit@ctrl-grav-02',\r\n                                      'rabbit@ctrl-stras-03']},\r\n              {'rabbit@ctrl-grav-02',['rabbit@ctrl-roub-01',\r\n                                      'rabbit@ctrl-stras-03']}]}]\r\n```\r\n\r\nTo fix this issue i tried to restart the rabbitmq one by one and got stuck on one node.\r\n\r\n```\r\nroot@ctrl-grav-02:~# rabbitmqctl stop_app\r\nStopping node 'rabbit@ctrl-grav-02' ...\r\n^C\r\nSession terminated, terminating shell...\r\nBREAK: (a)bort (c)ontinue (p)roc info (i)nfo (l)oaded\r\n       (v)ersion (k)ill (D)b-tables (d)istribution\r\n ...killed.\r\nroot@ctrl-grav-02:~# service rabbitmq-server restart\r\n * Restarting message broker rabbitmq-server                                                                                                                                                                                                  ^C^C^C                                                                                                                                                                                                                                 [fail]\r\nroot@ctrl-grav-02:~# \r\n```\r\n\r\nReset the node and join the node back to the cluster (for 2 nodes):\r\n```\r\nroot@ctrl-grav-02:~# rabbitmqctl join_cluster rabbit@ctrl-roub-01\r\nClustering node 'rabbit@ctrl-grav-02' with 'rabbit@ctrl-roub-01' ...\r\n...done (already_member).\r\nroot@ctrl-grav-02:~# rabbitmqctl start_app\r\nStarting node 'rabbit@ctrl-grav-02' ...\r\n...done.\r\n```\r\n\r\nRestarting all the ovs-workers and the webgui works again.\r\n\r\nI have no idea what the impact is for running vms...\r\nCreating vms will not work (nothing will be processed to put extra info into arakoon)\r\n\r\nAnyone an idea how we get partition problems?\r\nIs this network related cause previous night OVH did some maintenance on the network."},"comment":{"url":"https://api.github.com/repos/openvstorage/framework/issues/comments/243462084","html_url":"https://github.com/openvstorage/framework/issues/860#issuecomment-243462084","issue_url":"https://api.github.com/repos/openvstorage/framework/issues/860","id":243462084,"user":{"login":"kinvaris","id":4190375,"avatar_url":"https://avatars.githubusercontent.com/u/4190375?v=3","gravatar_id":"","url":"https://api.github.com/users/kinvaris","html_url":"https://github.com/kinvaris","followers_url":"https://api.github.com/users/kinvaris/followers","following_url":"https://api.github.com/users/kinvaris/following{/other_user}","gists_url":"https://api.github.com/users/kinvaris/gists{/gist_id}","starred_url":"https://api.github.com/users/kinvaris/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kinvaris/subscriptions","organizations_url":"https://api.github.com/users/kinvaris/orgs","repos_url":"https://api.github.com/users/kinvaris/repos","events_url":"https://api.github.com/users/kinvaris/events{/privacy}","received_events_url":"https://api.github.com/users/kinvaris/received_events","type":"User","site_admin":false},"created_at":"2016-08-30T14:40:13Z","updated_at":"2016-08-30T14:40:13Z","body":"Also the healthcheck will detect if there are any problems with rabbitmq or celery (ovs-workers)"}}
{"actor":{"display_login":"kinvaris"}}