{"action":"closed","issue":{"url":"https://api.github.com/repos/christkv/mongodb-core/issues/108","repository_url":"https://api.github.com/repos/christkv/mongodb-core","labels_url":"https://api.github.com/repos/christkv/mongodb-core/issues/108/labels{/name}","comments_url":"https://api.github.com/repos/christkv/mongodb-core/issues/108/comments","events_url":"https://api.github.com/repos/christkv/mongodb-core/issues/108/events","html_url":"https://github.com/christkv/mongodb-core/issues/108","id":164565383,"number":108,"title":"Hanging queries for \"hanging\" mongos proxy","user":{"login":"geronime","id":1003724,"avatar_url":"https://avatars.githubusercontent.com/u/1003724?v=3","gravatar_id":"","url":"https://api.github.com/users/geronime","html_url":"https://github.com/geronime","followers_url":"https://api.github.com/users/geronime/followers","following_url":"https://api.github.com/users/geronime/following{/other_user}","gists_url":"https://api.github.com/users/geronime/gists{/gist_id}","starred_url":"https://api.github.com/users/geronime/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/geronime/subscriptions","organizations_url":"https://api.github.com/users/geronime/orgs","repos_url":"https://api.github.com/users/geronime/repos","events_url":"https://api.github.com/users/geronime/events{/privacy}","received_events_url":"https://api.github.com/users/geronime/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":17,"created_at":"2016-07-08T16:16:27Z","updated_at":"2016-07-28T18:23:58Z","closed_at":"2016-07-28T18:23:58Z","body":"With one of my projects we repeatedly experienced an issue with hanging queries whenever a specific network partitioning happened; resulting in hanging (but not closed!) sockets. The project is hosted on public cloud and we've experienced the issue repeatedly so I'd assume somebody else must have run into the same hiccup.\r\n\r\nOur goal was to be able to detect this sort of event and instead of having \"hanging\" requests (caused by underlying hanging mongodb queries) to immediately respond with an error from the app. We did not want to explicitly time each mongo query out, not only because the project is quite large and it would be a considerable development effort. We'd prefer having some system solution that would be able to recognise the hanging state and disconnect the troubled proxy. After all the application shouldn't have to handle something that could seem more of a driver's job.\r\n\r\nWe figured out a dirty way to **simulate the issue**:\r\n* have a small cluster set up with several mongos proxies involved\r\n* have a simple [test script](https://gist.github.com/geronime/a0362e6d02af74c88e927abaef286b9e) running\r\n* send SIGSTOP to one of the mongos\r\n* as a result queries start to hang\r\n* after sending SIGCONT the queries resume, only they took as long as the mongos process was asleep\r\n\r\nClose enough to the real-live partitioned network scenario. We found out:\r\n* `mongos.connectOptions.socketTimeoutMS` doesn't help if there is sufficient traffic on the app\r\n* using `maxTimeMS` also doesn't help as this times a query out on server side as far as I understand and it never gets to the server during the hanging state\r\n* the keep-alive is not helpful at all as the default keep-alive settings are completely unsuitable for this use, we don't want to configure them on system level and there is no way to configure them through the driver as Node.js socket actually [doesn't support](https://github.com/nodejs/node/issues/4048) it so far\r\n\r\n**The workaround**:\r\nIn the initial workaround, where we've been using mongoose, we introduced regular pings in combination with `db.bufferMaxEntries:0`. Each iteration we issued as many pings as the driver had connections in the pool, timing the whole bulk of parallel pings out. Whenever a timeout hit we just forcefully closed the mongoose connection, making mongoose an automatic reconnection for us.\r\nThis workaround has a few downsides:\r\n* the whole connection is dropped and reconnected instead of disconnecting just the troubled proxy\r\n* issuing as many pings as all connections across all mongos proxies is a bit overkill - it is enough to ping just once per mongos; whenever the connection is stale, all of the connections in that particular proxy's pool behave the same way\r\n\r\nRecently we've been moving some parts of the app to use the mongodb driver instead of mongoose and therefore we had to come up with some non-mongoose solution, favourably without monkey-patching in every part of the app that is connecting to the database.\r\n\r\nI noticed that mongodb 2.1.x behaves a bit weird: I can see occasional reconnections of individual pool connections. This isn't just a result of the default `socketTimeoutMS=120000` that was introduced. I haven't looked deeper; in mongos logs I can see the test script dropping the connection and reconnecting later. With 2.0 this is not the case. mongodb@2.0 is using mongodb-core@1.2.\r\n\r\n**Solution?**\r\nI [forked](https://github.com/geronime/mongodb-core) mongodb-core and implemented custom keep-alive utilising pings - [diff against V1.2.32](https://github.com/geronime/mongodb-core/compare/V1.2.32...1.2.32_mongos_ping). I don't think this is an ideal solution as I haven't dug deep enough into the driver and it definitely could use some functional test simulation first. I'd be glad if you looked at it and gave me your feedback - perhaps, after some iterations, it would make sense to include something like this into the core driver or implement something else whatsoever to deal with the issue. \r\nThe extra mongos options would have to be propagated to the mongodb driver as well, for testing you can just uncomment the [appropriate line](https://github.com/geronime/mongodb-core/commit/0569f5d698221d3f5b066da15a62cfdda886fe98#diff-934e557209bbdb06743917230f540f3aR312).\r\n\r\nSorry for such a long article. :-)\r\n"}}
{"actor":{"display_login":"christkv"}}