{"action":"created","issue":{"url":"https://api.github.com/repos/mongodb-labs/mongo-connector/issues/549","repository_url":"https://api.github.com/repos/mongodb-labs/mongo-connector","labels_url":"https://api.github.com/repos/mongodb-labs/mongo-connector/issues/549/labels{/name}","comments_url":"https://api.github.com/repos/mongodb-labs/mongo-connector/issues/549/comments","events_url":"https://api.github.com/repos/mongodb-labs/mongo-connector/issues/549/events","html_url":"https://github.com/mongodb-labs/mongo-connector/issues/549","id":182276068,"number":549,"title":"mongo-connector stops working when active mongos and node go offline","user":{"login":"vimuston","id":22169004,"avatar_url":"https://avatars.githubusercontent.com/u/22169004?v=3","gravatar_id":"","url":"https://api.github.com/users/vimuston","html_url":"https://github.com/vimuston","followers_url":"https://api.github.com/users/vimuston/followers","following_url":"https://api.github.com/users/vimuston/following{/other_user}","gists_url":"https://api.github.com/users/vimuston/gists{/gist_id}","starred_url":"https://api.github.com/users/vimuston/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vimuston/subscriptions","organizations_url":"https://api.github.com/users/vimuston/orgs","repos_url":"https://api.github.com/users/vimuston/repos","events_url":"https://api.github.com/users/vimuston/events{/privacy}","received_events_url":"https://api.github.com/users/vimuston/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/mongodb-labs/mongo-connector/labels/bug","name":"bug","color":"fc2929"}],"state":"open","locked":false,"assignee":{"login":"ShaneHarvey","id":5015933,"avatar_url":"https://avatars.githubusercontent.com/u/5015933?v=3","gravatar_id":"","url":"https://api.github.com/users/ShaneHarvey","html_url":"https://github.com/ShaneHarvey","followers_url":"https://api.github.com/users/ShaneHarvey/followers","following_url":"https://api.github.com/users/ShaneHarvey/following{/other_user}","gists_url":"https://api.github.com/users/ShaneHarvey/gists{/gist_id}","starred_url":"https://api.github.com/users/ShaneHarvey/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ShaneHarvey/subscriptions","organizations_url":"https://api.github.com/users/ShaneHarvey/orgs","repos_url":"https://api.github.com/users/ShaneHarvey/repos","events_url":"https://api.github.com/users/ShaneHarvey/events{/privacy}","received_events_url":"https://api.github.com/users/ShaneHarvey/received_events","type":"User","site_admin":false},"assignees":[{"login":"ShaneHarvey","id":5015933,"avatar_url":"https://avatars.githubusercontent.com/u/5015933?v=3","gravatar_id":"","url":"https://api.github.com/users/ShaneHarvey","html_url":"https://github.com/ShaneHarvey","followers_url":"https://api.github.com/users/ShaneHarvey/followers","following_url":"https://api.github.com/users/ShaneHarvey/following{/other_user}","gists_url":"https://api.github.com/users/ShaneHarvey/gists{/gist_id}","starred_url":"https://api.github.com/users/ShaneHarvey/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ShaneHarvey/subscriptions","organizations_url":"https://api.github.com/users/ShaneHarvey/orgs","repos_url":"https://api.github.com/users/ShaneHarvey/repos","events_url":"https://api.github.com/users/ShaneHarvey/events{/privacy}","received_events_url":"https://api.github.com/users/ShaneHarvey/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/mongodb-labs/mongo-connector/milestones/6","html_url":"https://github.com/mongodb-labs/mongo-connector/milestones/2.5.0","labels_url":"https://api.github.com/repos/mongodb-labs/mongo-connector/milestones/6/labels","id":1890639,"number":6,"title":"2.5.0","description":"Feature release","creator":{"login":"llvtt","id":5168837,"avatar_url":"https://avatars.githubusercontent.com/u/5168837?v=3","gravatar_id":"","url":"https://api.github.com/users/llvtt","html_url":"https://github.com/llvtt","followers_url":"https://api.github.com/users/llvtt/followers","following_url":"https://api.github.com/users/llvtt/following{/other_user}","gists_url":"https://api.github.com/users/llvtt/gists{/gist_id}","starred_url":"https://api.github.com/users/llvtt/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/llvtt/subscriptions","organizations_url":"https://api.github.com/users/llvtt/orgs","repos_url":"https://api.github.com/users/llvtt/repos","events_url":"https://api.github.com/users/llvtt/events{/privacy}","received_events_url":"https://api.github.com/users/llvtt/received_events","type":"User","site_admin":false},"open_issues":8,"closed_issues":12,"state":"open","created_at":"2016-07-18T20:58:48Z","updated_at":"2016-10-25T22:34:23Z","due_on":null,"closed_at":null},"comments":4,"created_at":"2016-10-11T14:17:46Z","updated_at":"2016-10-26T14:11:16Z","closed_at":null,"body":"Our environment has following MongoDB cluster setup:\r\n**Site 1:**\r\nconfig1: config & mongos\r\nconfig2: config & mongos\r\nnode1: rs1 & rs2\r\nnode2: rs1 & rs2\r\n\r\n**Site 3:**\r\nconfig3: config & mongos\r\nnode3: rs1 & rs2\r\n\r\nmongo-connector **mainAddress** is \"config1:27017,config2:27017,config3:27017\"\r\n\r\nOn the weekend **Site 1** went down and node3 on **Site 2** was elected primary. During this time mongo-connector raised an AutoReconnect and stopped indexing documents. The mongo-connector process is still alive, but nothing is getting inserted to Solr.\r\n\r\nJudging from where the exception was raised from, it would seem that the OplogThread noticed the disconnection and the control returned to Connector. However, the mongos Connector was connected to (this I am guessing but it would make sense) is also down, because it resides on the same site. In this scenario, the main connection should probably be reconnected?\r\n\r\n[mongo-connector.txt](https://github.com/mongodb-labs/mongo-connector/files/521814/mongo-connector.txt)\r\n\r\nEdit: Including the traceback:\r\n```python\r\n2016-10-09 00:53:06,512 [ERROR] mongo_connector.util:92 - Fatal Exception\r\nTraceback (most recent call last):\r\n  File \"/usr/lib/python2.6/site-packages/mongo_connector-2.4.1-py2.6.egg/mongo_connector/util.py\", line 90, in wrapped\r\n    func(*args, **kwargs)\r\n  File \"/usr/lib/python2.6/site-packages/mongo_connector-2.4.1-py2.6.egg/mongo_connector/connector.py\", line 316, in run\r\n    for shard_doc in main_conn['config']['shards'].find():\r\n  File \"build/bdist.linux-x86_64/egg/pymongo/cursor.py\", line 1149, in next\r\n    if len(self.__data) or self._refresh():\r\n  File \"build/bdist.linux-x86_64/egg/pymongo/cursor.py\", line 1081, in _refresh\r\n    self.__codec_options.uuid_representation))\r\n  File \"build/bdist.linux-x86_64/egg/pymongo/cursor.py\", line 996, in __send_message\r\n    res = client._send_message_with_response(message, **kwargs)\r\n  File \"build/bdist.linux-x86_64/egg/pymongo/mongo_client.py\", line 1366, in _send_message_with_response\r\n    raise AutoReconnect(str(e))\r\nAutoReconnect: [Errno 113] No route to host\r\n```"},"comment":{"url":"https://api.github.com/repos/mongodb-labs/mongo-connector/issues/comments/256359292","html_url":"https://github.com/mongodb-labs/mongo-connector/issues/549#issuecomment-256359292","issue_url":"https://api.github.com/repos/mongodb-labs/mongo-connector/issues/549","id":256359292,"user":{"login":"vimuston","id":22169004,"avatar_url":"https://avatars.githubusercontent.com/u/22169004?v=3","gravatar_id":"","url":"https://api.github.com/users/vimuston","html_url":"https://github.com/vimuston","followers_url":"https://api.github.com/users/vimuston/followers","following_url":"https://api.github.com/users/vimuston/following{/other_user}","gists_url":"https://api.github.com/users/vimuston/gists{/gist_id}","starred_url":"https://api.github.com/users/vimuston/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vimuston/subscriptions","organizations_url":"https://api.github.com/users/vimuston/orgs","repos_url":"https://api.github.com/users/vimuston/repos","events_url":"https://api.github.com/users/vimuston/events{/privacy}","received_events_url":"https://api.github.com/users/vimuston/received_events","type":"User","site_admin":false},"created_at":"2016-10-26T14:11:16Z","updated_at":"2016-10-26T14:11:16Z","body":"@ShaneHarvey Are you currently working on this? I noticed #561 seems to fix a similar kind of issue. I have a fix proposal for the connector.py part: wrap the 'config.shards' loop in try/except and catch pymongo.errors.AutoReconnect, so that the next connection attempt reconnects correctly. I tested it in our environment by shutting down the active mongos and one of the oplog mongod instances. mongo-connector seemed to recover correctly. "}}
{"actor":{"display_login":"vimuston"}}