{"action":"closed","issue":{"url":"https://api.github.com/repos/mozilla/fxa-auth-db-mysql/issues/108","repository_url":"https://api.github.com/repos/mozilla/fxa-auth-db-mysql","labels_url":"https://api.github.com/repos/mozilla/fxa-auth-db-mysql/issues/108/labels{/name}","comments_url":"https://api.github.com/repos/mozilla/fxa-auth-db-mysql/issues/108/comments","events_url":"https://api.github.com/repos/mozilla/fxa-auth-db-mysql/issues/108/events","html_url":"https://github.com/mozilla/fxa-auth-db-mysql/issues/108","id":108771026,"number":108,"title":"Pool does not reconnect on db failover","user":{"login":"rfk","id":34695,"avatar_url":"https://avatars.githubusercontent.com/u/34695?v=3","gravatar_id":"","url":"https://api.github.com/users/rfk","html_url":"https://github.com/rfk","followers_url":"https://api.github.com/users/rfk/followers","following_url":"https://api.github.com/users/rfk/following{/other_user}","gists_url":"https://api.github.com/users/rfk/gists{/gist_id}","starred_url":"https://api.github.com/users/rfk/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rfk/subscriptions","organizations_url":"https://api.github.com/users/rfk/orgs","repos_url":"https://api.github.com/users/rfk/repos","events_url":"https://api.github.com/users/rfk/events{/privacy}","received_events_url":"https://api.github.com/users/rfk/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/mozilla/fxa-auth-db-mysql/labels/waffle:backlog","name":"waffle:backlog","color":"ededed"}],"state":"closed","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/mozilla/fxa-auth-db-mysql/milestones/12","html_url":"https://github.com/mozilla/fxa-auth-db-mysql/milestones/FxA-0:%20quality","labels_url":"https://api.github.com/repos/mozilla/fxa-auth-db-mysql/milestones/12/labels","id":1265647,"number":12,"title":"FxA-0: quality","description":"An ongoing milestone into which we can schedule maintenance tasks, tech-debt paydown, and other code-quality-related issues.","creator":{"login":"rfk","id":34695,"avatar_url":"https://avatars.githubusercontent.com/u/34695?v=3","gravatar_id":"","url":"https://api.github.com/users/rfk","html_url":"https://github.com/rfk","followers_url":"https://api.github.com/users/rfk/followers","following_url":"https://api.github.com/users/rfk/following{/other_user}","gists_url":"https://api.github.com/users/rfk/gists{/gist_id}","starred_url":"https://api.github.com/users/rfk/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rfk/subscriptions","organizations_url":"https://api.github.com/users/rfk/orgs","repos_url":"https://api.github.com/users/rfk/repos","events_url":"https://api.github.com/users/rfk/events{/privacy}","received_events_url":"https://api.github.com/users/rfk/received_events","type":"User","site_admin":false},"open_issues":0,"closed_issues":4,"state":"open","created_at":"2015-08-21T02:49:44Z","updated_at":"2016-06-09T06:09:57Z","due_on":null,"closed_at":null},"comments":12,"created_at":"2015-09-29T00:12:49Z","updated_at":"2016-06-09T06:09:57Z","closed_at":"2016-06-09T06:09:57Z","body":"From email conversation with @jrgm:\r\n\r\n\"\"\"\r\nI was testing RDS Multi-AZ failover in stage under load on Friday. \r\n\r\nThe TL;DR is that fxa-auth does not reconnect on failover. Ugh.\r\n\r\nfxa-oauth and fxa-profile did reconnect, and on a restart fxa-auth\r\nreconnects. But on 3 of 4 occasions, fxa-auth wound up in a state where it\r\nkept putting queries in the queue and returning 500 on 'Error: Queue limit\r\nreached'.\r\n\"\"\"\r\n\r\nWe do some mucking around with the error-recovery options, which may be affecting this.  \r\nFrom https://github.com/mozilla/fxa-auth-db-mysql/blob/master/lib/db/mysql.js#L34:\r\n```\r\n    // poolCluster will remove the pool after `removeNodeErrorCount` errors.\r\n    // We don't ever want to remove a pool because we only have one pool\r\n    // for writing and reading each. Connection errors are mostly out of our\r\n    // control for automatic recovery so monitoring of 503s is critical.\r\n    // Since `removeNodeErrorCount` is Infinity `canRetry` must be false\r\n    // to prevent inifinite retry attempts.\r\n    this.poolCluster = mysql.createPoolCluster(\r\n      {\r\n        removeNodeErrorCount: Infinity,\r\n        canRetry: false\r\n      }\r\n    )\r\n```\r\n\r\n"}}
{"actor":{"display_login":"jrgm"}}