{"action":"created","issue":{"url":"https://api.github.com/repos/helm/charts/issues/10032","repository_url":"https://api.github.com/repos/helm/charts","labels_url":"https://api.github.com/repos/helm/charts/issues/10032/labels{/name}","comments_url":"https://api.github.com/repos/helm/charts/issues/10032/comments","events_url":"https://api.github.com/repos/helm/charts/issues/10032/events","html_url":"https://github.com/helm/charts/pull/10032","id":391309401,"node_id":"MDExOlB1bGxSZXF1ZXN0MjM4ODY2NTMx","number":10032,"title":"[stable/redis-ha] Fix split brain issue","user":{"login":"ssalaues","id":24439848,"node_id":"MDQ6VXNlcjI0NDM5ODQ4","avatar_url":"https://avatars1.githubusercontent.com/u/24439848?v=4","gravatar_id":"","url":"https://api.github.com/users/ssalaues","html_url":"https://github.com/ssalaues","followers_url":"https://api.github.com/users/ssalaues/followers","following_url":"https://api.github.com/users/ssalaues/following{/other_user}","gists_url":"https://api.github.com/users/ssalaues/gists{/gist_id}","starred_url":"https://api.github.com/users/ssalaues/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ssalaues/subscriptions","organizations_url":"https://api.github.com/users/ssalaues/orgs","repos_url":"https://api.github.com/users/ssalaues/repos","events_url":"https://api.github.com/users/ssalaues/events{/privacy}","received_events_url":"https://api.github.com/users/ssalaues/received_events","type":"User","site_admin":false},"labels":[{"id":1037076588,"node_id":"MDU6TGFiZWwxMDM3MDc2NTg4","url":"https://api.github.com/repos/helm/charts/labels/Contribution%20Allowed","name":"Contribution Allowed","color":"3df47d","default":false},{"id":676596499,"node_id":"MDU6TGFiZWw2NzY1OTY0OTk=","url":"https://api.github.com/repos/helm/charts/labels/size/L","name":"size/L","color":"ee9900","default":false}],"state":"open","locked":false,"assignee":{"login":"cpanato","id":4115580,"node_id":"MDQ6VXNlcjQxMTU1ODA=","avatar_url":"https://avatars2.githubusercontent.com/u/4115580?v=4","gravatar_id":"","url":"https://api.github.com/users/cpanato","html_url":"https://github.com/cpanato","followers_url":"https://api.github.com/users/cpanato/followers","following_url":"https://api.github.com/users/cpanato/following{/other_user}","gists_url":"https://api.github.com/users/cpanato/gists{/gist_id}","starred_url":"https://api.github.com/users/cpanato/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cpanato/subscriptions","organizations_url":"https://api.github.com/users/cpanato/orgs","repos_url":"https://api.github.com/users/cpanato/repos","events_url":"https://api.github.com/users/cpanato/events{/privacy}","received_events_url":"https://api.github.com/users/cpanato/received_events","type":"User","site_admin":false},"assignees":[{"login":"cpanato","id":4115580,"node_id":"MDQ6VXNlcjQxMTU1ODA=","avatar_url":"https://avatars2.githubusercontent.com/u/4115580?v=4","gravatar_id":"","url":"https://api.github.com/users/cpanato","html_url":"https://github.com/cpanato","followers_url":"https://api.github.com/users/cpanato/followers","following_url":"https://api.github.com/users/cpanato/following{/other_user}","gists_url":"https://api.github.com/users/cpanato/gists{/gist_id}","starred_url":"https://api.github.com/users/cpanato/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cpanato/subscriptions","organizations_url":"https://api.github.com/users/cpanato/orgs","repos_url":"https://api.github.com/users/cpanato/repos","events_url":"https://api.github.com/users/cpanato/events{/privacy}","received_events_url":"https://api.github.com/users/cpanato/received_events","type":"User","site_admin":false}],"milestone":null,"comments":4,"created_at":"2018-12-14T23:09:56Z","updated_at":"2018-12-20T22:42:20Z","closed_at":null,"author_association":"CONTRIBUTOR","pull_request":{"url":"https://api.github.com/repos/helm/charts/pulls/10032","html_url":"https://github.com/helm/charts/pull/10032","diff_url":"https://github.com/helm/charts/pull/10032.diff","patch_url":"https://github.com/helm/charts/pull/10032.patch"},"body":"#### What this PR does / why we need it:\r\nMajor:\r\nFixes issues where network partitions can result in a split-brain #9488 \r\n  - Better init script that better handles recovery from the split-brain\r\n  - Retained Sentinel IDs\r\n  - Better probes\r\n\r\nMinor:\r\nFixes name override options #9568\r\n\r\n---\r\nPart of the root issue of #9488 in that whenever a pod is deleted/recreated, it gets assigned a new IP address causing the remaining quorum to lose track of the down pod after rescheduling. In ideal scenarios this is okay because the init container will find and attach to an existing quorum. However in the case of a network partition this can easily result in the split-brain issue.\r\n\r\nThese changes leverage the stable IP of a Kubernetes service mapped to the notion of an \"Announce IP\" in redis/sentinel that allow for more stable operation and discovery. This is necessary because both Redis and Sentinel will resolve DNS names and only persist the corresponding IP in the config not the DNS name. \r\n\r\n#### Which issue this PR fixes\r\n*(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*\r\n  - fixes #9568, fixes #9488\r\n\r\n#### Special notes for your reviewer:\r\nIn my test lab, these fixes allow for the the init to fail and crash, preventing the issue from occurring.\r\n\r\ncc @aianus \r\n\r\n#### Checklist\r\n[Place an '[x]' (no spaces) in all applicable fields. Please remove unrelated fields.]\r\n- [x] [DCO](https://www.helm.sh/blog/helm-dco/index.html) signed\r\n- [x] Chart Version bumped\r\n- [x] Variables are documented in the README.md"},"comment":{"url":"https://api.github.com/repos/helm/charts/issues/comments/449159339","html_url":"https://github.com/helm/charts/pull/10032#issuecomment-449159339","issue_url":"https://api.github.com/repos/helm/charts/issues/10032","id":449159339,"node_id":"MDEyOklzc3VlQ29tbWVudDQ0OTE1OTMzOQ==","user":{"login":"burnettk","id":18027,"node_id":"MDQ6VXNlcjE4MDI3","avatar_url":"https://avatars3.githubusercontent.com/u/18027?v=4","gravatar_id":"","url":"https://api.github.com/users/burnettk","html_url":"https://github.com/burnettk","followers_url":"https://api.github.com/users/burnettk/followers","following_url":"https://api.github.com/users/burnettk/following{/other_user}","gists_url":"https://api.github.com/users/burnettk/gists{/gist_id}","starred_url":"https://api.github.com/users/burnettk/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/burnettk/subscriptions","organizations_url":"https://api.github.com/users/burnettk/orgs","repos_url":"https://api.github.com/users/burnettk/repos","events_url":"https://api.github.com/users/burnettk/events{/privacy}","received_events_url":"https://api.github.com/users/burnettk/received_events","type":"User","site_admin":false},"created_at":"2018-12-20T22:42:20Z","updated_at":"2018-12-20T22:42:20Z","author_association":"NONE","body":"@ssalaues @aianus i just noticed this PR and the corresponding issue. we also filed https://github.com/helm/charts/pull/10178, which i think may address the issue. we took a very different approach. curious to see what you all think. we referenced an acceptance test script in the PR that you may find useful. ultimately, we're sad that we have 10 to 20-second \"downtimes\" during rolling deployments (as can be seen with the acceptance test script) with the current chart, but really happy that some of the edge cases that cause much worse effects are being driven out. thanks!"}}
{"actor":{"display_login":"burnettk"}}