{"action":"created","issue":{"url":"https://api.github.com/repos/openvstorage/framework/issues/1199","repository_url":"https://api.github.com/repos/openvstorage/framework","labels_url":"https://api.github.com/repos/openvstorage/framework/issues/1199/labels{/name}","comments_url":"https://api.github.com/repos/openvstorage/framework/issues/1199/comments","events_url":"https://api.github.com/repos/openvstorage/framework/issues/1199/events","html_url":"https://github.com/openvstorage/framework/issues/1199","id":191224753,"number":1199,"title":"ovs remove node does not work on node that is offline","user":{"login":"kinvaris","id":4190375,"avatar_url":"https://avatars.githubusercontent.com/u/4190375?v=3","gravatar_id":"","url":"https://api.github.com/users/kinvaris","html_url":"https://github.com/kinvaris","followers_url":"https://api.github.com/users/kinvaris/followers","following_url":"https://api.github.com/users/kinvaris/following{/other_user}","gists_url":"https://api.github.com/users/kinvaris/gists{/gist_id}","starred_url":"https://api.github.com/users/kinvaris/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kinvaris/subscriptions","organizations_url":"https://api.github.com/users/kinvaris/orgs","repos_url":"https://api.github.com/users/kinvaris/repos","events_url":"https://api.github.com/users/kinvaris/events{/privacy}","received_events_url":"https://api.github.com/users/kinvaris/received_events","type":"User","site_admin":false},"labels":[{"id":361467358,"url":"https://api.github.com/repos/openvstorage/framework/labels/priority_urgent","name":"priority_urgent","color":"d93f0b","default":false},{"id":339407574,"url":"https://api.github.com/repos/openvstorage/framework/labels/state_inprogress","name":"state_inprogress","color":"c2e0c6","default":false},{"id":239189394,"url":"https://api.github.com/repos/openvstorage/framework/labels/type_bug","name":"type_bug","color":"fef2c0","default":false}],"state":"open","locked":false,"assignee":{"login":"khenderick","id":1453214,"avatar_url":"https://avatars.githubusercontent.com/u/1453214?v=3","gravatar_id":"","url":"https://api.github.com/users/khenderick","html_url":"https://github.com/khenderick","followers_url":"https://api.github.com/users/khenderick/followers","following_url":"https://api.github.com/users/khenderick/following{/other_user}","gists_url":"https://api.github.com/users/khenderick/gists{/gist_id}","starred_url":"https://api.github.com/users/khenderick/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/khenderick/subscriptions","organizations_url":"https://api.github.com/users/khenderick/orgs","repos_url":"https://api.github.com/users/khenderick/repos","events_url":"https://api.github.com/users/khenderick/events{/privacy}","received_events_url":"https://api.github.com/users/khenderick/received_events","type":"User","site_admin":false},"assignees":[{"login":"khenderick","id":1453214,"avatar_url":"https://avatars.githubusercontent.com/u/1453214?v=3","gravatar_id":"","url":"https://api.github.com/users/khenderick","html_url":"https://github.com/khenderick","followers_url":"https://api.github.com/users/khenderick/followers","following_url":"https://api.github.com/users/khenderick/following{/other_user}","gists_url":"https://api.github.com/users/khenderick/gists{/gist_id}","starred_url":"https://api.github.com/users/khenderick/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/khenderick/subscriptions","organizations_url":"https://api.github.com/users/khenderick/orgs","repos_url":"https://api.github.com/users/khenderick/repos","events_url":"https://api.github.com/users/khenderick/events{/privacy}","received_events_url":"https://api.github.com/users/khenderick/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/openvstorage/framework/milestones/8","html_url":"https://github.com/openvstorage/framework/milestones/Fargo%20RC2","labels_url":"https://api.github.com/repos/openvstorage/framework/milestones/8/labels","id":2111518,"number":8,"title":"Fargo RC2","description":"","creator":{"login":"wimpers","id":8392927,"avatar_url":"https://avatars.githubusercontent.com/u/8392927?v=3","gravatar_id":"","url":"https://api.github.com/users/wimpers","html_url":"https://github.com/wimpers","followers_url":"https://api.github.com/users/wimpers/followers","following_url":"https://api.github.com/users/wimpers/following{/other_user}","gists_url":"https://api.github.com/users/wimpers/gists{/gist_id}","starred_url":"https://api.github.com/users/wimpers/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wimpers/subscriptions","organizations_url":"https://api.github.com/users/wimpers/orgs","repos_url":"https://api.github.com/users/wimpers/repos","events_url":"https://api.github.com/users/wimpers/events{/privacy}","received_events_url":"https://api.github.com/users/wimpers/received_events","type":"User","site_admin":false},"open_issues":21,"closed_issues":12,"state":"open","created_at":"2016-11-03T08:26:35Z","updated_at":"2016-11-23T15:59:58Z","due_on":"2016-12-16T00:00:00Z","closed_at":null},"comments":3,"created_at":"2016-11-23T09:43:31Z","updated_at":"2016-11-24T08:30:20Z","closed_at":null,"body":"While performing the `ovs remove node` on a node that was offline:\r\n```\r\n2016-11-23 10:30:24 09300 +0100 - ctrl-roub-01 - 109786/140116299626240 - lib/setup - 382 - INFO - Demote complete\r\n2016-11-23 10:30:24 09300 +0100 - ctrl-roub-01 - 109786/140116299626240 - lib/setup - 383 - INFO -     Stopping and removing services\r\n2016-11-23 10:30:24 09300 +0100 - ctrl-roub-01 - 109786/140116299626240 - lib/setup - 384 - INFO -     Removing node from model\r\n2016-11-23 10:30:24 10000 +0100 - ctrl-roub-01 - 109786/140116299626240 - lib/scheduled tasks - 385 - INFO - Send data task scheduling started\r\n2016-11-23 10:30:24 11200 +0100 - ctrl-roub-01 - 109786/140116299626240 - lib/setup - 386 - INFO - Running \"remove\" hooks\r\n2016-11-23 10:30:24 11200 +0100 - ctrl-roub-01 - 109786/140116299626240 - lib/setup - 387 - INFO - Executing storagedriver.on_remove\r\n2016-11-23 10:30:26 42000 +0100 - ctrl-roub-01 - 109786/140116299626240 - lib/setup - 388 - INFO - \r\n\r\n2016-11-23 10:30:26 42100 +0100 - ctrl-roub-01 - 109786/140116299626240 - lib/setup - 389 - ERROR - An unexpected error occurred:\r\nTraceback (most recent call last):\r\n  File \"ovs/lib/setup.py\", line 743, in remove_node\r\n    SetupController._run_hooks('remove', storage_router_to_remove.ip, complete_removal=remove_asd_manager)\r\n  File \"ovs/lib/setup.py\", line 1814, in _run_hooks\r\n    function(cluster_ip=cluster_ip, **kwargs)\r\n  File \"ovs/lib/helpers/..//storagedriver.py\", line 136, in on_remove\r\n    client = SSHClient(endpoint=cluster_ip, username='root')\r\n  File \"ovs/extensions/generic/sshclient.py\", line 152, in __init__\r\n    self._connect()\r\n  File \"ovs/extensions/generic/sshclient.py\", line 173, in _connect\r\n    self._client.connect(self.ip, username=self.username, password=self.password)\r\n  File \"/usr/lib/python2.7/dist-packages/paramiko/client.py\", line 311, in connect\r\n    raise NoValidConnectionsError(errors)\r\nNoValidConnectionsError: [Errno None] Unable to connect to port 22 on  or 172.20.20.7\r\n2016-11-23 10:30:26 42200 +0100 - ctrl-roub-01 - 109786/140116299626240 - lib/setup - 390 - ERROR - [Errno None] Unable to connect to port 22 on  or 172.20.20.7\r\nTraceback (most recent call last):\r\n  File \"ovs/lib/setup.py\", line 743, in remove_node\r\n    SetupController._run_hooks('remove', storage_router_to_remove.ip, complete_removal=remove_asd_manager)\r\n  File \"ovs/lib/setup.py\", line 1814, in _run_hooks\r\n    function(cluster_ip=cluster_ip, **kwargs)\r\n  File \"ovs/lib/helpers/..//storagedriver.py\", line 136, in on_remove\r\n    client = SSHClient(endpoint=cluster_ip, username='root')\r\n  File \"ovs/extensions/generic/sshclient.py\", line 152, in __init__\r\n    self._connect()\r\n  File \"ovs/extensions/generic/sshclient.py\", line 173, in _connect\r\n    self._client.connect(self.ip, username=self.username, password=self.password)\r\n  File \"/usr/lib/python2.7/dist-packages/paramiko/client.py\", line 311, in connect\r\n    raise NoValidConnectionsError(errors)\r\nNoValidConnectionsError: [Errno None] Unable to connect to port 22 on  or 172.20.20.7\r\n```\r\n\r\nAfter activating the nodes again, I reran the `ovs remove node` and it worked."},"comment":{"url":"https://api.github.com/repos/openvstorage/framework/issues/comments/262717656","html_url":"https://github.com/openvstorage/framework/issues/1199#issuecomment-262717656","issue_url":"https://api.github.com/repos/openvstorage/framework/issues/1199","id":262717656,"user":{"login":"khenderick","id":1453214,"avatar_url":"https://avatars.githubusercontent.com/u/1453214?v=3","gravatar_id":"","url":"https://api.github.com/users/khenderick","html_url":"https://github.com/khenderick","followers_url":"https://api.github.com/users/khenderick/followers","following_url":"https://api.github.com/users/khenderick/following{/other_user}","gists_url":"https://api.github.com/users/khenderick/gists{/gist_id}","starred_url":"https://api.github.com/users/khenderick/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/khenderick/subscriptions","organizations_url":"https://api.github.com/users/khenderick/orgs","repos_url":"https://api.github.com/users/khenderick/repos","events_url":"https://api.github.com/users/khenderick/events{/privacy}","received_events_url":"https://api.github.com/users/khenderick/received_events","type":"User","site_admin":false},"created_at":"2016-11-24T08:30:19Z","updated_at":"2016-11-24T08:30:19Z","body":"Wrapped exception to expected one\r\n\r\n```\r\nIn [1]: from ovs.extensions.generic.sshclient import SSHClient\r\n\r\nIn [2]: c = SSHClient('10.100.169.101')\r\n---------------------------------------------------------------------------\r\nUnableToConnectException                  Traceback (most recent call last)\r\n<ipython-input-2-a20bc2e22991> in <module>()\r\n----> 1 c = SSHClient('10.100.169.101')\r\n\r\n/opt/OpenvStorage/ovs/extensions/generic/sshclient.py in __init__(self, endpoint, username, password)\r\n    152                 SSHClient.client_cache[key] = client\r\n    153             self._client = SSHClient.client_cache[key]\r\n--> 154         self._connect()\r\n    155 \r\n    156     def __del__(self):\r\n\r\n/opt/OpenvStorage/ovs/extensions/generic/sshclient.py in _connect(self)\r\n    187             if 'No route to host' in message or 'Unable to connect' in message:\r\n    188                 SSHClient._logger.error(message)\r\n--> 189                 raise UnableToConnectException(message)\r\n    190             raise\r\n    191 \r\n\r\nUnableToConnectException: [Errno None] Unable to connect to port 22 on  or 10.100.169.101\r\n```"}}
{"actor":{"display_login":"khenderick"}}