{"action":"created","issue":{"url":"https://api.github.com/repos/brianc/node-postgres/issues/1942","repository_url":"https://api.github.com/repos/brianc/node-postgres","labels_url":"https://api.github.com/repos/brianc/node-postgres/issues/1942/labels{/name}","comments_url":"https://api.github.com/repos/brianc/node-postgres/issues/1942/comments","events_url":"https://api.github.com/repos/brianc/node-postgres/issues/1942/events","html_url":"https://github.com/brianc/node-postgres/issues/1942","id":477360625,"node_id":"MDU6SXNzdWU0NzczNjA2MjU=","number":1942,"title":"node-postgres handles server disconnection differently on macOS and Linux","user":{"login":"jmacmahon","id":1281068,"node_id":"MDQ6VXNlcjEyODEwNjg=","avatar_url":"https://avatars0.githubusercontent.com/u/1281068?v=4","gravatar_id":"","url":"https://api.github.com/users/jmacmahon","html_url":"https://github.com/jmacmahon","followers_url":"https://api.github.com/users/jmacmahon/followers","following_url":"https://api.github.com/users/jmacmahon/following{/other_user}","gists_url":"https://api.github.com/users/jmacmahon/gists{/gist_id}","starred_url":"https://api.github.com/users/jmacmahon/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jmacmahon/subscriptions","organizations_url":"https://api.github.com/users/jmacmahon/orgs","repos_url":"https://api.github.com/users/jmacmahon/repos","events_url":"https://api.github.com/users/jmacmahon/events{/privacy}","received_events_url":"https://api.github.com/users/jmacmahon/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2019-08-06T12:48:28Z","updated_at":"2019-08-06T15:07:52Z","closed_at":null,"author_association":"NONE","body":"Hello, the company I work for has been using `pg-promise` for our database service and we've run into an issue with failover, which we believe is an error in the underlying `node-postgres` library. We believe the issue is due to the way Linux handles socket timeout events differently to macOS.\r\n\r\n### Steps to reproduce:\r\n- Connect to a Postgres server. We used one hosted in AWS RDS with MultiAZ failover enabled.\r\n- Run a query every N seconds\r\n- Reboot the server such that the connection is dropped without a TCP FIN packet. We did a reboot with failover in AWS RDS.\r\n\r\nNote: we believe this scenario is not specific to RDS, but rather any network outage or server failure which does not send a TCP FIN packet.\r\n\r\n### Expected outcome and actual outcome on macOS:\r\n- The next query fails and the failing client is removed from the pool.\r\n- Subsequent queries use a new client which tries to establish a fresh connection.\r\n- When the server reboot/failover is complete, these queries will succeed.\r\n\r\n### Actual outcome on Linux:\r\n- The next query fails, but the bad client is not removed from the pool.\r\n- Subsequent queries try to re-use the bad client and fail even after the reboot/failover is complete.\r\n\r\n### Detailed order of events\r\n#### macOS\r\n- Successful query\r\n- Reboot DB\r\n- DB stops listening on original IP\r\n- Client begins a further query\r\n- TCP sends query, does not recieve an ACK\r\n- TCP begins retransmission, does not receive an ACK\r\n- Approximately 18 seconds after sending, TCP ceases retransmission and sends a RST\r\n- Client rejects query promise with \"Error: read ETIMEDOUT\"\r\n- Immediately after the error the \"connection\" remains in node-pg's pool\r\n- Almost immediately afterwards the pool emits an \"error\" event\r\n- The \"connection\" is removed from node-pg's pool\r\n- Client begins another query\r\n- DNS fetches the new IP\r\n- TCP successfully submits and retrieves the query to the new IP\r\n\r\n#### Linux\r\n- Successful query\r\n- Reboot DB\r\n- DB stops listening on original IP\r\n- Client begins a further query\r\n- TCP sends query, does not receive an ACK\r\n- TCP begins retransmission, does not receive an ACK\r\n- Approximately 18 seconds after sending, TCP ceases retransmission and sends a RST\r\n- Client rejects query promise with a \"Error: Connection terminated unexpectedly\"\r\n    ```\r\n    error Error: Connection terminated unexpectedly\r\n      at Connection.con.once (/src/node_modules/pg-promise/node_modules/pg/lib/client.js:235:9)\r\n      at Object.onceWrapper (events.js:313:30)\r\n      at emitNone (events.js:106:13)\r\n      at Connection.emit (events.js:208:7)\r\n      at Socket.<anonymous> (/src/node_modules/pg-promise/node_modules/pg/lib/connection.js:131:10)\r\n      at emitNone (events.js:111:20)\r\n      at Socket.emit (events.js:208:7)\r\n      at endReadableNT (_stream_readable.js:1056:12)\r\n      at _combinedTickCallback (internal/process/next_tick.js:138:11)\r\n      at process._tickDomainCallback (internal/process/next_tick.js:218:9)\r\n    ```\r\n- After the error, the \"connection\" remains in node-pg's pool \r\n- Subsequent queries fail immediately without sending any data with the following error:\r\n    ```\r\n    error Error: Client has encountered a connection error and is not queryable\r\n      at process.nextTick (/src/node_modules/pg-promise/node_modules/pg/lib/client.js:500:25)\r\n      at _combinedTickCallback (internal/process/next_tick.js:131:7)\r\n      at process._tickDomainCallback (internal/process/next_tick.js:218:9)\r\n    ```"},"comment":{"url":"https://api.github.com/repos/brianc/node-postgres/issues/comments/518711544","html_url":"https://github.com/brianc/node-postgres/issues/1942#issuecomment-518711544","issue_url":"https://api.github.com/repos/brianc/node-postgres/issues/1942","id":518711544,"node_id":"MDEyOklzc3VlQ29tbWVudDUxODcxMTU0NA==","user":{"login":"charmander","id":1889843,"node_id":"MDQ6VXNlcjE4ODk4NDM=","avatar_url":"https://avatars1.githubusercontent.com/u/1889843?v=4","gravatar_id":"","url":"https://api.github.com/users/charmander","html_url":"https://github.com/charmander","followers_url":"https://api.github.com/users/charmander/followers","following_url":"https://api.github.com/users/charmander/following{/other_user}","gists_url":"https://api.github.com/users/charmander/gists{/gist_id}","starred_url":"https://api.github.com/users/charmander/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/charmander/subscriptions","organizations_url":"https://api.github.com/users/charmander/orgs","repos_url":"https://api.github.com/users/charmander/repos","events_url":"https://api.github.com/users/charmander/events{/privacy}","received_events_url":"https://api.github.com/users/charmander/received_events","type":"User","site_admin":false},"created_at":"2019-08-06T15:07:52Z","updated_at":"2019-08-06T15:07:52Z","author_association":"COLLABORATOR","body":"The pool removes clients pretty aggressively (e.g. even when the query error is an error in the query). Are you using `pool.query` or acquiring clients manually? What does the error handling look like there?"}}
{"actor":{"display_login":"charmander"}}