{"action":"created","issue":{"url":"https://api.github.com/repos/dnaeon/py-vpoller/issues/194","repository_url":"https://api.github.com/repos/dnaeon/py-vpoller","labels_url":"https://api.github.com/repos/dnaeon/py-vpoller/issues/194/labels{/name}","comments_url":"https://api.github.com/repos/dnaeon/py-vpoller/issues/194/comments","events_url":"https://api.github.com/repos/dnaeon/py-vpoller/issues/194/events","html_url":"https://github.com/dnaeon/py-vpoller/issues/194","id":161661313,"number":194,"title":"host unavailability cause whole vpoller-worker to fail","user":{"login":"jabalv","id":2405761,"avatar_url":"https://avatars.githubusercontent.com/u/2405761?v=3","gravatar_id":"","url":"https://api.github.com/users/jabalv","html_url":"https://github.com/jabalv","followers_url":"https://api.github.com/users/jabalv/followers","following_url":"https://api.github.com/users/jabalv/following{/other_user}","gists_url":"https://api.github.com/users/jabalv/gists{/gist_id}","starred_url":"https://api.github.com/users/jabalv/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jabalv/subscriptions","organizations_url":"https://api.github.com/users/jabalv/orgs","repos_url":"https://api.github.com/users/jabalv/repos","events_url":"https://api.github.com/users/jabalv/events{/privacy}","received_events_url":"https://api.github.com/users/jabalv/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2016-06-22T11:49:19Z","updated_at":"2016-06-29T08:45:38Z","closed_at":"2016-06-27T07:13:17Z","body":"When I monitor esxi hosts and one of them fails or become unreachable from worker logs:\r\n\r\n> ERROR/VPollerWorker-1] Cannot connect to hostname: [Errno 110] Connection timed out\r\n\r\nor\r\n\r\n> Cannot connect to hostname: [Errno 113] No route to host\r\n\r\nThen whole vpoller stops and do not perform any checks, but processes still show heatlhy status and are running. No more records in log files.\r\n\r\nErrors messages:\r\n\r\n> [2016-06-22 14:27:00,958 - WARNING/VPollerWorker-2] Task host.perf.metric.get failed: Traceback (most recent call last):\r\n>   File \"/usr/lib/python2.7/site-packages/vpoller/task/decorators.py\", line 55, in wrapper\r\n>     result = fn(*args, **kwargs)\r\n>   File \"/usr/lib/python2.7/site-packages/vpoller/vsphere/tasks.py\", line 1462, in host_perf_metric_get\r\n>     obj_type=pyVmomi.vim.HostSystem\r\n>   File \"/usr/lib/python2.7/site-packages/vconnector/core.py\", line 412, in get_object_by_property\r\n>     view_ref = self.get_container_view(obj_type=[obj_type])\r\n>   File \"/usr/lib/python2.7/site-packages/vconnector/core.py\", line 347, in get_container_view\r\n>     container = self.si.content.rootFolder\r\n>   File \"/usr/lib/python2.7/site-packages/vconnector/core.py\", line 123, in si\r\n>     self.connect()\r\n>   File \"/usr/lib/python2.7/site-packages/vconnector/core.py\", line 159, in connect\r\n>     sslContext=self.ssl_context,\r\n>   File \"/usr/lib/python2.7/site-packages/pyVim/connect.py\", line 786, in SmartConnect\r\n>     sslContext)\r\n>   File \"/usr/lib/python2.7/site-packages/pyVim/connect.py\", line 671, in __FindSupportedVersion\r\n>     sslContext)\r\n>   File \"/usr/lib/python2.7/site-packages/pyVim/connect.py\", line 591, in __GetServiceVersionDescription\r\n>     path + \"/vimServiceVersions.xml\", sslContext)\r\n>   File \"/usr/lib/python2.7/site-packages/pyVim/connect.py\", line 557, in __GetElementTree\r\n>     conn.request(\"GET\", path)\r\n>   File \"/usr/lib64/python2.7/httplib.py\", line 979, in request\r\n>     self._send_request(method, url, body, headers)\r\n>   File \"/usr/lib64/python2.7/httplib.py\", line 1013, in _send_request\r\n>     self.endheaders(body)\r\n>   File \"/usr/lib64/python2.7/httplib.py\", line 975, in endheaders\r\n>     self._send_output(message_body)\r\n>   File \"/usr/lib64/python2.7/httplib.py\", line 835, in _send_output\r\n>     self.send(msg)\r\n>   File \"/usr/lib64/python2.7/httplib.py\", line 797, in send\r\n>     self.connect()\r\n>   File \"/usr/lib64/python2.7/httplib.py\", line 1189, in connect\r\n>     HTTPConnection.connect(self)\r\n>   File \"/usr/lib64/python2.7/httplib.py\", line 778, in connect\r\n>     self.timeout, self.source_address)\r\n>   File \"/usr/lib64/python2.7/socket.py\", line 571, in create_connection\r\n>     raise err\r\n> error: [Errno 110] Connection timed out\r\n\r\n> [2016-06-22 14:14:22,056 - WARNING/VPollerWorker-1] Task host.perf.metric.get failed: Traceback (most recent call last):\r\n>   File \"/usr/lib/python2.7/site-packages/vpoller/task/decorators.py\", line 55, in wrapper\r\n>     result = fn(*args, **kwargs)\r\n>   File \"/usr/lib/python2.7/site-packages/vpoller/vsphere/tasks.py\", line 1462, in host_perf_metric_get\r\n>     obj_type=pyVmomi.vim.HostSystem\r\n>   File \"/usr/lib/python2.7/site-packages/vconnector/core.py\", line 412, in get_object_by_property\r\n>     view_ref = self.get_container_view(obj_type=[obj_type])\r\n>   File \"/usr/lib/python2.7/site-packages/vconnector/core.py\", line 347, in get_container_view\r\n>     container = self.si.content.rootFolder\r\n>   File \"/usr/lib/python2.7/site-packages/vconnector/core.py\", line 123, in si\r\n>     self.connect()\r\n>   File \"/usr/lib/python2.7/site-packages/vconnector/core.py\", line 159, in connect\r\n>     sslContext=self.ssl_context,\r\n>   File \"/usr/lib/python2.7/site-packages/pyVim/connect.py\", line 786, in SmartConnect\r\n>     sslContext)\r\n>   File \"/usr/lib/python2.7/site-packages/pyVim/connect.py\", line 671, in __FindSupportedVersion\r\n>     sslContext)\r\n>   File \"/usr/lib/python2.7/site-packages/pyVim/connect.py\", line 591, in __GetServiceVersionDescription\r\n>     path + \"/vimServiceVersions.xml\", sslContext)\r\n>   File \"/usr/lib/python2.7/site-packages/pyVim/connect.py\", line 557, in __GetElementTree\r\n>     conn.request(\"GET\", path)\r\n>   File \"/usr/lib64/python2.7/httplib.py\", line 979, in request\r\n>     self._send_request(method, url, body, headers)\r\n>   File \"/usr/lib64/python2.7/httplib.py\", line 1013, in _send_request\r\n>     self.endheaders(body)\r\n>   File \"/usr/lib64/python2.7/httplib.py\", line 975, in endheaders\r\n>     self._send_output(message_body)\r\n>   File \"/usr/lib64/python2.7/httplib.py\", line 835, in _send_output\r\n>     self.send(msg)\r\n>   File \"/usr/lib64/python2.7/httplib.py\", line 797, in send\r\n>     self.connect()\r\n>   File \"/usr/lib64/python2.7/httplib.py\", line 1189, in connect\r\n>     HTTPConnection.connect(self)\r\n>   File \"/usr/lib64/python2.7/httplib.py\", line 778, in connect\r\n>     self.timeout, self.source_address)\r\n>   File \"/usr/lib64/python2.7/socket.py\", line 571, in create_connection\r\n>     raise err\r\n> error: [Errno 113] No route to host\r\n\r\nI was able to solve issue only by removing failed hosts from database.\r\nvpoller-worker --version\r\n0.6.1\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/dnaeon/py-vpoller/issues/comments/229294867","html_url":"https://github.com/dnaeon/py-vpoller/issues/194#issuecomment-229294867","issue_url":"https://api.github.com/repos/dnaeon/py-vpoller/issues/194","id":229294867,"user":{"login":"jabalv","id":2405761,"avatar_url":"https://avatars.githubusercontent.com/u/2405761?v=3","gravatar_id":"","url":"https://api.github.com/users/jabalv","html_url":"https://github.com/jabalv","followers_url":"https://api.github.com/users/jabalv/followers","following_url":"https://api.github.com/users/jabalv/following{/other_user}","gists_url":"https://api.github.com/users/jabalv/gists{/gist_id}","starred_url":"https://api.github.com/users/jabalv/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jabalv/subscriptions","organizations_url":"https://api.github.com/users/jabalv/orgs","repos_url":"https://api.github.com/users/jabalv/repos","events_url":"https://api.github.com/users/jabalv/events{/privacy}","received_events_url":"https://api.github.com/users/jabalv/received_events","type":"User","site_admin":false},"created_at":"2016-06-29T08:45:38Z","updated_at":"2016-06-29T08:45:38Z","body":"I still think that vPoller should not crash/stuck if hosts is not reachable? Maybe there could implement some check before creating agents from vconnect-cli side or when connection to hosts is lost, it will not stuck but update/mark host in sqlite db. I think this should be investigated, but I need some time..."}}
{"actor":{"display_login":"jabalv"}}