{"action":"created","issue":{"url":"https://api.github.com/repos/mongodb-labs/mongo-connector/issues/791","repository_url":"https://api.github.com/repos/mongodb-labs/mongo-connector","labels_url":"https://api.github.com/repos/mongodb-labs/mongo-connector/issues/791/labels{/name}","comments_url":"https://api.github.com/repos/mongodb-labs/mongo-connector/issues/791/comments","events_url":"https://api.github.com/repos/mongodb-labs/mongo-connector/issues/791/events","html_url":"https://github.com/mongodb-labs/mongo-connector/issues/791","id":293431638,"number":791,"title":"Data loss if elasticsearch is not running","user":{"login":"VitaliStupin","id":15190150,"avatar_url":"https://avatars3.githubusercontent.com/u/15190150?v=4","gravatar_id":"","url":"https://api.github.com/users/VitaliStupin","html_url":"https://github.com/VitaliStupin","followers_url":"https://api.github.com/users/VitaliStupin/followers","following_url":"https://api.github.com/users/VitaliStupin/following{/other_user}","gists_url":"https://api.github.com/users/VitaliStupin/gists{/gist_id}","starred_url":"https://api.github.com/users/VitaliStupin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/VitaliStupin/subscriptions","organizations_url":"https://api.github.com/users/VitaliStupin/orgs","repos_url":"https://api.github.com/users/VitaliStupin/repos","events_url":"https://api.github.com/users/VitaliStupin/events{/privacy}","received_events_url":"https://api.github.com/users/VitaliStupin/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-02-01T07:38:27Z","updated_at":"2018-03-07T05:48:34Z","closed_at":null,"author_association":"NONE","body":"If elasticsearch service is not running when mongo-connector fetches new data from mongodb then:\r\n\r\n1. A warning is printed to mongo-connector.log:\r\n```\r\n2018-02-01 09:03:25,524 [WARNING] elasticsearch:97 - GET http://localhost:9200/_mget?realtime=true [status:N/A request:0.000s]\r\nTraceback (most recent call last):\r\n  File \"/usr/local/lib/python2.7/dist-packages/elasticsearch/connection/http_urllib3.py\", line 147, in perform_request\r\n    response = self.pool.urlopen(method, url, body, retries=False, headers=self.headers, **kw)\r\n  File \"/usr/local/lib/python2.7/dist-packages/urllib3/connectionpool.py\", line 639, in urlopen\r\n    _stacktrace=sys.exc_info()[2])\r\n  File \"/usr/local/lib/python2.7/dist-packages/urllib3/util/retry.py\", line 333, in increment\r\n    raise six.reraise(type(error), error, _stacktrace)\r\n  File \"/usr/local/lib/python2.7/dist-packages/urllib3/connectionpool.py\", line 601, in urlopen\r\n    chunked=chunked)\r\n  File \"/usr/local/lib/python2.7/dist-packages/urllib3/connectionpool.py\", line 357, in _make_request\r\n    conn.request(method, url, **httplib_request_kw)\r\n  File \"/usr/lib/python2.7/httplib.py\", line 1057, in request\r\n    self._send_request(method, url, body, headers)\r\n  File \"/usr/lib/python2.7/httplib.py\", line 1097, in _send_request\r\n    self.endheaders(body)\r\n  File \"/usr/lib/python2.7/httplib.py\", line 1053, in endheaders\r\n    self._send_output(message_body)\r\n  File \"/usr/lib/python2.7/httplib.py\", line 897, in _send_output\r\n    self.send(msg)\r\n  File \"/usr/lib/python2.7/httplib.py\", line 859, in send\r\n    self.connect()\r\n  File \"/usr/local/lib/python2.7/dist-packages/urllib3/connection.py\", line 166, in connect\r\n    conn = self._new_conn()\r\n  File \"/usr/local/lib/python2.7/dist-packages/urllib3/connection.py\", line 150, in _new_conn\r\n    self, \"Failed to establish a new connection: %s\" % e)\r\nNewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f19cc4b1f10>: Failed to establish a new connection: [Errno 111] Connection refused\r\n```\r\n\r\n2. mongo-connector will not exit and will not retry to reconnect to elasticsearch after some time, but instead would run in a broken state. Effectively replication stops until mongo-connector is restarted.\r\n\r\n3. new timestamp would be written to oplog.timestamp as if the data was successfully replicated.\r\n\r\n4. after restart of mongo-connector (if elasticsearch service gets online again) mongo-connector would skip the data it failed to write before, because oplog.timestamp already points to a newer timestamp.\r\n\r\nA desired behavior could be either exiting mongo-connector without updating oplog.timestamp or retrying the connection to elasticsearch after some predefined intervals.\r\n\r\nThis problem may appear for example in situations when both elasticsearch and mongo-connector are started during system startup and elasticsearch is not ready yet to receive connection when mongo-connector already fetched some data from mongodb. As a workaround it is possible to add a delay to mongo-connector startup, but that would not help in a situation when elasticsearch service is restarting.\r\n\r\nUsed software:\r\n\r\nPython (2.7.12):\r\nmongo-connector (2.5.1)\r\npymongo (3.6.0)\r\nelasticsearch (6.0.0)\r\nelastic2-doc-manager (0.3.0)\r\n\r\nElasticsearch 6.0.0\r\nMongoDB 3.4.10\r\nUbuntu 16.04\r\n\r\nconfig.json:\r\n```\r\n{\r\n    \"mainAddress\": \"mongodb:27017\",\r\n    \"authentication\": {\r\n        \"adminUsername\": \"admin\",\r\n        \"password\": \"password\"\r\n    },\r\n    \"namespaces\": {\r\n        \"xxx.data\": \"yyy.data\"\r\n    },\r\n    \"docManagers\": [\r\n        {\r\n            \"docManager\": \"elastic2_doc_manager\",\r\n            \"targetURL\": \"localhost:9200\"\r\n        }\r\n    ]\r\n}\r\n```"},"comment":{"url":"https://api.github.com/repos/mongodb-labs/mongo-connector/issues/comments/371030342","html_url":"https://github.com/mongodb-labs/mongo-connector/issues/791#issuecomment-371030342","issue_url":"https://api.github.com/repos/mongodb-labs/mongo-connector/issues/791","id":371030342,"user":{"login":"kgrvamsi","id":1199031,"avatar_url":"https://avatars3.githubusercontent.com/u/1199031?v=4","gravatar_id":"","url":"https://api.github.com/users/kgrvamsi","html_url":"https://github.com/kgrvamsi","followers_url":"https://api.github.com/users/kgrvamsi/followers","following_url":"https://api.github.com/users/kgrvamsi/following{/other_user}","gists_url":"https://api.github.com/users/kgrvamsi/gists{/gist_id}","starred_url":"https://api.github.com/users/kgrvamsi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kgrvamsi/subscriptions","organizations_url":"https://api.github.com/users/kgrvamsi/orgs","repos_url":"https://api.github.com/users/kgrvamsi/repos","events_url":"https://api.github.com/users/kgrvamsi/events{/privacy}","received_events_url":"https://api.github.com/users/kgrvamsi/received_events","type":"User","site_admin":false},"created_at":"2018-03-07T05:48:34Z","updated_at":"2018-03-07T05:48:34Z","author_association":"NONE","body":"I was thinking the same on how the Elasticsearch failure scenario is addressed and would expect some sort of parameter where it can resync with the data on a couple of retries\r\n\r\n```\r\nmongo-connector -v --resync-check \r\n```"}}
{"actor":{"display_login":"kgrvamsi"}}