{"action":"created","issue":{"url":"https://api.github.com/repos/CodisLabs/codis/issues/1462","repository_url":"https://api.github.com/repos/CodisLabs/codis","labels_url":"https://api.github.com/repos/CodisLabs/codis/issues/1462/labels{/name}","comments_url":"https://api.github.com/repos/CodisLabs/codis/issues/1462/comments","events_url":"https://api.github.com/repos/CodisLabs/codis/issues/1462/events","html_url":"https://github.com/CodisLabs/codis/issues/1462","id":306376447,"number":1462,"title":"Codis-HA的使用问题","user":{"login":"YrlixJoe","id":31159377,"avatar_url":"https://avatars3.githubusercontent.com/u/31159377?v=4","gravatar_id":"","url":"https://api.github.com/users/YrlixJoe","html_url":"https://github.com/YrlixJoe","followers_url":"https://api.github.com/users/YrlixJoe/followers","following_url":"https://api.github.com/users/YrlixJoe/following{/other_user}","gists_url":"https://api.github.com/users/YrlixJoe/gists{/gist_id}","starred_url":"https://api.github.com/users/YrlixJoe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/YrlixJoe/subscriptions","organizations_url":"https://api.github.com/users/YrlixJoe/orgs","repos_url":"https://api.github.com/users/YrlixJoe/repos","events_url":"https://api.github.com/users/YrlixJoe/events{/privacy}","received_events_url":"https://api.github.com/users/YrlixJoe/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2018-03-19T08:52:40Z","updated_at":"2018-03-20T02:07:07Z","closed_at":null,"author_association":"NONE","body":"在使用codis-ha时出现如下问题，不知是否正常。\r\n\r\n**场景如下**：\r\n启用codis-ha后，当一个组内的master节点down掉后，codis-ha会选举group内的一个slave节点作为master节点。**但随即会剔除group内除新提升为master的节点以外，其他所有节点都会被强制剔除且服务器上确认codis-server进程退出。** 当手动启动被剔除的节点后，再次添加到该group依然无法添加（添加后会被自动剔除且进程自动退出）。\r\n\r\n**此外，想要了解一些问题**：\r\n1. 想要了解codis-ha的实现机制，是基于redis-sentinel实现的吗？\r\n2. codis-ha在使用时，需要另外以sentinel的模式运行redis吗？\r\n3. codis-ha的主从切换是怎样的？\r\n4. 后续的数据同步的流程（当原master重启上线再加入到group中时）是怎样的呢？\r\n\r\n\r\n**望解答，谢谢。**\r\n\r\n----\r\n\r\n#### 相关信息如下\r\n**group组实例**\r\n![snipaste_20180319_163447.jpg](https://i.loli.net/2018/03/19/5aaf76c082303.jpg)\r\n\r\n6181为master，6282为slave。\r\n\r\n**Codis-server日志信息**\r\n6381\r\n> 8541:signal-handler (1521448920) Received SIGTERM scheduling shutdown...\r\n8541:M 19 Mar 16:42:00.622 # User requested shutdown...\r\n8541:M 19 Mar 16:42:00.622 * Removing the pid file.\r\n8541:M 19 Mar 16:42:00.623 # Redis is now ready to exit, bye bye...\r\n\r\n\r\n6282\r\n>1002:S 19 Mar 16:42:00.625 # Connection with master lost.\r\n1002:S 19 Mar 16:42:00.626 * Caching the disconnected master state.\r\n1002:S 19 Mar 16:42:00.630 * Connecting to MASTER 10.1.60.72:6381\r\n1002:S 19 Mar 16:42:00.631 * MASTER <-> SLAVE sync started\r\n1002:S 19 Mar 16:42:00.631 # Error condition on socket for SYNC: Connection refused\r\n1002:S 19 Mar 16:42:01.632 * Connecting to MASTER 10.1.60.72:6381\r\n1002:S 19 Mar 16:42:01.632 * MASTER <-> SLAVE sync started\r\n1002:S 19 Mar 16:42:01.632 # Error condition on socket for SYNC: Connection refused\r\n1002:M 19 Mar 16:42:01.729 * Discarding previously cached master state.\r\n1002:M 19 Mar 16:42:01.730 * MASTER MODE enabled (user request from 'id=307 addr=10.1.60.71:49794 fd=5 name= age=0 idle=0 flags=x db=0 sub=0 psub=0 multi=4 qbuf=0 qbuf-free=32768 obl=50 oll=0 omem=0 events=r cmd=exec')\r\n1002:M 19 Mar 16:42:01.730 # CONFIG REWRITE executed with success.\r\n\r\n\r\n\r\n**Codis-HA日志如下（日志级别Debug）**\r\n\r\n> 2018/03/19 16:42:01 api.go:139: [DEBUG] call rpc [GET] http://10.1.60.71:18080/api/topom/stats/95b62887719520f17e312eaa76d28f2b in 12.590253ms\r\n2018/03/19 16:42:01 main.go:206: [INFO] [ ] proxy-1 [T] 73c66af68de27c782306597171dd8a35 [A] 10.1.60.71:11080 [P] 10.1.60.71:19000\r\n2018/03/19 16:42:01 main.go:206: [INFO] [ ] proxy-2 [T] 0fd8a1488d9b366dac8e09ac5235e4c2 [A] 10.1.60.72:11080 [P] 10.1.60.72:19000\r\n2018/03/19 16:42:01 main.go:206: [INFO] [ ] proxy-3 [T] e3619878e11ed86d6c0d378f9b0a918e [A] 10.1.60.73:11080 [P] 10.1.60.73:19000\r\n2018/03/19 16:42:01 main.go:240: [INFO] [ ] group-1 [0] 10.1.60.71:6379\r\n2018/03/19 16:42:01 main.go:240: [INFO] [ ] group-1 [1] 10.1.60.71:6380\r\n2018/03/19 16:42:01 main.go:240: [INFO] [ ] group-2 [0] 10.1.60.72:6379\r\n2018/03/19 16:42:01 main.go:240: [INFO] [ ] group-2 [1] 10.1.60.72:6380\r\n2018/03/19 16:42:01 main.go:240: [INFO] [ ] group-3 [0] 10.1.60.73:6379\r\n2018/03/19 16:42:01 main.go:240: [INFO] [ ] group-3 [1] 10.1.60.73:6380\r\n2018/03/19 16:42:01 main.go:236: [WARN] [E] group-4 [0] 10.1.60.72:6381\r\n2018/03/19 16:42:01 main.go:242: [WARN] [X] group-4 [1] 10.1.60.72:6382\r\n2018/03/19 16:42:01 main.go:274: [WARN] codis-server (master) 10.1.60.72:6381 state error\r\n2018/03/19 16:42:01 main.go:360: [WARN] try to promote group-[4] with slave 10.1.60.72:6382\r\n2018/03/19 16:42:01 api.go:139: [DEBUG] call rpc [PUT] http://10.1.60.71:18080/api/topom/group/promote/95b62887719520f17e312eaa76d28f2b/4/10.1.60.72:6382 in 31.323133ms\r\n2018/03/19 16:42:01 main.go:364: [WARN] done.\r\n\r\n\r\n> 2018/03/19 16:42:04 api.go:139: [DEBUG] call rpc [GET] http://10.1.60.71:18080/api/topom/stats/95b62887719520f17e312eaa76d28f2b in 13.616047ms\r\n2018/03/19 16:42:04 main.go:206: [INFO] [ ] proxy-1 [T] 73c66af68de27c782306597171dd8a35 [A] 10.1.60.71:11080 [P] 10.1.60.71:19000\r\n2018/03/19 16:42:04 main.go:206: [INFO] [ ] proxy-2 [T] 0fd8a1488d9b366dac8e09ac5235e4c2 [A] 10.1.60.72:11080 [P] 10.1.60.72:19000\r\n2018/03/19 16:42:04 main.go:206: [INFO] [ ] proxy-3 [T] e3619878e11ed86d6c0d378f9b0a918e [A] 10.1.60.73:11080 [P] 10.1.60.73:19000\r\n2018/03/19 16:42:04 main.go:240: [INFO] [ ] group-1 [0] 10.1.60.71:6379\r\n2018/03/19 16:42:04 main.go:240: [INFO] [ ] group-1 [1] 10.1.60.71:6380\r\n2018/03/19 16:42:04 main.go:240: [INFO] [ ] group-2 [0] 10.1.60.72:6379\r\n2018/03/19 16:42:04 main.go:240: [INFO] [ ] group-2 [1] 10.1.60.72:6380\r\n2018/03/19 16:42:04 main.go:240: [INFO] [ ] group-3 [0] 10.1.60.73:6379\r\n2018/03/19 16:42:04 main.go:240: [INFO] [ ] group-3 [1] 10.1.60.73:6380\r\n2018/03/19 16:42:04 main.go:240: [INFO] [ ] group-4 [0] 10.1.60.72:6382\r\n2018/03/19 16:42:04 main.go:236: [WARN] [E] group-4 [1] 10.1.60.72:6381\r\n2018/03/19 16:42:04 main.go:286: [WARN] try to group-del-server to dashboard 10.1.60.72:6381\r\n2018/03/19 16:42:04 api.go:139: [DEBUG] call rpc [PUT] http://10.1.60.71:18080/api/topom/group/del/95b62887719520f17e312eaa76d28f2b/4/10.1.60.72:6381 in 2.467563ms\r\n2018/03/19 16:42:04 main.go:291: [DEBUG] call rpc group-del-server OK\r\n2018/03/19 16:42:04 main.go:294: [WARN] try to shutdown codis-server(slave) 10.1.60.72:6381\r\n2018/03/19 16:42:04 main.go:297: [WARN] connect to codis-server(slave) 10.1.60.72:6381 failed\r\n[error]: dial tcp 10.1.60.72:6381: getsockopt: connection refused\r\n    2   /usr/local/codis/src/github.com/CodisLabs/codis/pkg/utils/redis/client.go:42\r\n            github.com/CodisLabs/codis/pkg/utils/redis.NewClient\r\n    1   /usr/local/codis/src/github.com/CodisLabs/codis/cmd/ha/main.go:295\r\n            main.(*HealthyChecker).Maintains\r\n    0   /usr/local/codis/src/github.com/CodisLabs/codis/cmd/ha/main.go:95\r\n            main.main\r\n        ... ...\r\n"},"comment":{"url":"https://api.github.com/repos/CodisLabs/codis/issues/comments/374447845","html_url":"https://github.com/CodisLabs/codis/issues/1462#issuecomment-374447845","issue_url":"https://api.github.com/repos/CodisLabs/codis/issues/1462","id":374447845,"user":{"login":"YrlixJoe","id":31159377,"avatar_url":"https://avatars3.githubusercontent.com/u/31159377?v=4","gravatar_id":"","url":"https://api.github.com/users/YrlixJoe","html_url":"https://github.com/YrlixJoe","followers_url":"https://api.github.com/users/YrlixJoe/followers","following_url":"https://api.github.com/users/YrlixJoe/following{/other_user}","gists_url":"https://api.github.com/users/YrlixJoe/gists{/gist_id}","starred_url":"https://api.github.com/users/YrlixJoe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/YrlixJoe/subscriptions","organizations_url":"https://api.github.com/users/YrlixJoe/orgs","repos_url":"https://api.github.com/users/YrlixJoe/repos","events_url":"https://api.github.com/users/YrlixJoe/events{/privacy}","received_events_url":"https://api.github.com/users/YrlixJoe/received_events","type":"User","site_admin":false},"created_at":"2018-03-20T02:07:07Z","updated_at":"2018-03-20T02:07:07Z","author_association":"NONE","body":"@spinlock 谢谢，可否帮忙解答提的几个问题。"}}
{"actor":{"display_login":"YrlixJoe"}}