{"action":"opened","issue":{"url":"https://api.github.com/repos/antirez/redis/issues/3306","repository_url":"https://api.github.com/repos/antirez/redis","labels_url":"https://api.github.com/repos/antirez/redis/issues/3306/labels{/name}","comments_url":"https://api.github.com/repos/antirez/redis/issues/3306/comments","events_url":"https://api.github.com/repos/antirez/redis/issues/3306/events","html_url":"https://github.com/antirez/redis/issues/3306","id":159564417,"number":3306,"title":"Sentinel failover abort in network partition","user":{"login":"mayuraviraj","id":4223352,"avatar_url":"https://avatars.githubusercontent.com/u/4223352?v=3","gravatar_id":"","url":"https://api.github.com/users/mayuraviraj","html_url":"https://github.com/mayuraviraj","followers_url":"https://api.github.com/users/mayuraviraj/followers","following_url":"https://api.github.com/users/mayuraviraj/following{/other_user}","gists_url":"https://api.github.com/users/mayuraviraj/gists{/gist_id}","starred_url":"https://api.github.com/users/mayuraviraj/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mayuraviraj/subscriptions","organizations_url":"https://api.github.com/users/mayuraviraj/orgs","repos_url":"https://api.github.com/users/mayuraviraj/repos","events_url":"https://api.github.com/users/mayuraviraj/events{/privacy}","received_events_url":"https://api.github.com/users/mayuraviraj/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2016-06-10T05:59:18Z","updated_at":"2016-06-10T05:59:18Z","closed_at":null,"body":"My team experienced sentinel failover issue when doing network partition testing on Redis 3.0.5 version. Our redis setup as follows,\r\n\r\nOne master node, two replica nodes, and 3 sentinels to monitor the master. Sentinels have quorum of 2. \r\n\r\nWith above setup we tested for failover network partition in following manner.\r\n\r\nNetwork partition 1 :\r\nMaster node and 1 sentinel.\r\n\r\nNetwork partition 2 :\r\nTwo slave nodes and two sentinels. \r\n\r\nExpected outcome:\r\nIn case of network partition, which is defined in above, we expected to happen a failover in partition 2 since it match the requirement for partition. \r\n\r\nActual outcome:\r\nFailover never occurs in partition 2. Master in the setup is the original master which is in partition 1. \r\n\r\nNote;\r\nTo check the network partition first we modified the iptable such that above network isolation occurs. After observing this issue, to make sure partition occurs we tried partitioning by removing the network cable. This was done as following manner. We configured all the entities which should be in network partition 2 in one machine ( machine 1 ). i.e two slaves and two sentinels are configured in one machine. Then we configured other entities which should belong to partition 1 in different machine ( machine 2 ). i.e master and one sentinel. Then  we isolated machine 1 by it's network cable thus removing it connectivity from network, hoping this should trigger a failover in machine 1 ( partition 2). But this failover never happened. Following are the sentinel logs,\r\n\r\n**Network partition 1 sentinel log**\r\n5221:X 10 Jun 10:42:38.146 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.\r\n5221:X 10 Jun 10:42:38.147 # Sentinel runid is b7aa8a1240294586faf4844ae022d0f29deee525\r\n5221:X 10 Jun 10:42:38.147 # +monitor master mymaster master_host 6379 quorum 2\r\n5221:X 10 Jun 10:44:58.632 * +slave slave slave_host:6379 slave_host 6379 @ mymaster master_host 6379\r\n5221:X 10 Jun 10:46:08.919 * +slave slave slave_host:6380 slave_host 6380 @ mymaster master_host 6379\r\n5221:X 10 Jun 10:53:55.446 * +sentinel sentinel slave_host:26379 slave_host 26379 @ mymaster master_host 6379\r\n5221:X 10 Jun 10:54:29.702 * +sentinel sentinel slave_host:26380 slave_host 26380 @ mymaster master_host 6379\r\n5221:X 10 Jun 10:55:18.478 # +new-epoch 1\r\n5221:X 10 Jun 10:57:50.321 # +sdown sentinel slave_host:26380 slave_host 26380 @ mymaster master_host 6379\r\n5221:X 10 Jun 10:57:50.460 # +sdown slave slave_host:6380 slave_host 6380 @ mymaster master_host 6379\r\n5221:X 10 Jun 10:57:50.460 # +sdown slave slave_host:6379 slave_host 6379 @ mymaster master_host 6379\r\n5221:X 10 Jun 10:57:50.460 # +sdown sentinel slave_host:26379 slave_host 26379 @ mymaster master_host 6379\r\n5221:X 10 Jun 11:05:25.885 # +new-epoch 2\r\n5221:X 10 Jun 11:05:27.046 # -sdown slave slave_host:6380 slave_host 6380 @ mymaster master_host 6379\r\n5221:X 10 Jun 11:05:27.047 # -sdown slave slave_host:6379 slave_host 6379 @ mymaster master_host 6379\r\n5221:X 10 Jun 11:05:27.047 # -sdown sentinel slave_host:26380 slave_host 26380 @ mymaster master_host 6379\r\n5221:X 10 Jun 11:05:27.047 # -sdown sentinel slave_host:26379 slave_host 26379 @ mymaster master_host 6379\r\n\r\n\r\n**Network partition 2 sentinel 1 log**\r\n27769:X 10 Jun 11:30:52.710 # Sentinel runid is 806a125c3cb76dd4ee04d97b4a4b8c10377d7d57\r\n27769:X 10 Jun 11:30:52.710 # +monitor master mymaster slave_host 6379 quorum 2\r\n27769:X 10 Jun 11:30:54.452 * +sentinel sentinel master_host:26379 master_host 26379 @ mymaster slave_host 6379\r\n27769:X 10 Jun 11:31:28.970 * +sentinel sentinel slave_host:26380 slave_host 26380 @ mymaster slave_host 6379\r\n27769:X 10 Jun 11:31:42.712 # +sdown master mymaster slave_host 6379\r\n27769:X 10 Jun 11:32:17.095 # +new-epoch 1\r\n27769:X 10 Jun 11:32:17.137 # +vote-for-leader 3c7addf4c91bdcffb9d4c4eeea31e0037cac40bc 1\r\n27769:X 10 Jun 11:32:17.996 # +odown master mymaster slave_host 6379 #quorum 2/2\r\n27769:X 10 Jun 11:32:17.996 # Next failover delay: I will not start a failover before Fri Jun 10 11:38:17 2016\r\n27769:X 10 Jun 11:34:27.652 # -odown master mymaster slave_host 6379\r\n27769:X 10 Jun 11:34:48.913 # +sdown sentinel master_host:26379 master_host 26379 @ mymaster slave_host 6379\r\n27769:X 10 Jun 11:34:54.027 # +sdown sentinel slave_host:26380 slave_host 26380 @ mymaster slave_host 6379\r\n27769:X 10 Jun 11:42:24.142 # -sdown sentinel slave_host:26380 slave_host 26380 @ mymaster slave_host 6379\r\n27769:X 10 Jun 11:42:24.209 # +odown master mymaster slave_host 6379 #quorum 2/2\r\n27769:X 10 Jun 11:42:24.209 # +new-epoch 2\r\n27769:X 10 Jun 11:42:24.209 # +try-failover master mymaster slave_host 6379\r\n27769:X 10 Jun 11:42:24.264 # +vote-for-leader 806a125c3cb76dd4ee04d97b4a4b8c10377d7d57 2\r\n27769:X 10 Jun 11:42:24.264 # slave_host:26380 voted for 3c7addf4c91bdcffb9d4c4eeea31e0037cac40bc 2\r\n27769:X 10 Jun 11:42:25.215 # -sdown sentinel master_host:26379 master_host 26379 @ mymaster slave_host 6379\r\n27769:X 10 Jun 11:42:34.397 # -failover-abort-not-elected master mymaster slave_host 6379\r\n27769:X 10 Jun 11:42:34.473 # Next failover delay: I will not start a failover before Fri Jun 10 11:48:24 2016\r\n\r\n**Network partition 2 sentinel 2 log**\r\n27779:X 10 Jun 11:31:26.934 # Sentinel runid is 3c7addf4c91bdcffb9d4c4eeea31e0037cac40bc\r\n27779:X 10 Jun 11:31:26.934 # +monitor master mymaster slave_host 6379 quorum 2\r\n27779:X 10 Jun 11:31:27.254 * +sentinel sentinel slave_host:26379 slave_host 26379 @ mymaster slave_host 6379\r\n27779:X 10 Jun 11:31:27.560 * +sentinel sentinel master_host:26379 master_host 26379 @ mymaster slave_host 6379\r\n27779:X 10 Jun 11:32:16.937 # +sdown master mymaster slave_host 6379\r\n27779:X 10 Jun 11:32:17.003 # +odown master mymaster slave_host 6379 #quorum 2/2\r\n27779:X 10 Jun 11:32:17.003 # +new-epoch 1\r\n27779:X 10 Jun 11:32:17.003 # +try-failover master mymaster slave_host 6379\r\n27779:X 10 Jun 11:32:17.053 # +vote-for-leader 3c7addf4c91bdcffb9d4c4eeea31e0037cac40bc 1\r\n27779:X 10 Jun 11:32:17.137 # slave_host:26379 voted for 3c7addf4c91bdcffb9d4c4eeea31e0037cac40bc 1\r\n27779:X 10 Jun 11:32:17.194 # +elected-leader master mymaster slave_host 6379\r\n27779:X 10 Jun 11:32:17.194 # +failover-state-select-slave master mymaster slave_host 6379\r\n27779:X 10 Jun 11:32:17.265 # -failover-abort-no-good-slave master mymaster slave_host 6379\r\n27779:X 10 Jun 11:32:17.317 # Next failover delay: I will not start a failover before Fri Jun 10 11:38:17 2016\r\n27779:X 10 Jun 11:34:27.514 # -odown master mymaster slave_host 6379\r\n27779:X 10 Jun 11:34:49.321 # +sdown sentinel master_host:26379 master_host 26379 @ mymaster slave_host 6379\r\n27779:X 10 Jun 11:34:54.181 # +sdown sentinel slave_host:26379 slave_host 26379 @ mymaster slave_host 6379\r\n27779:X 10 Jun 11:42:24.122 # -sdown sentinel slave_host:26379 slave_host 26379 @ mymaster slave_host 6379\r\n27779:X 10 Jun 11:42:24.178 # +odown master mymaster slave_host 6379 #quorum 2/2\r\n27779:X 10 Jun 11:42:24.178 # +new-epoch 2\r\n27779:X 10 Jun 11:42:24.178 # +try-failover master mymaster slave_host 6379\r\n27779:X 10 Jun 11:42:24.222 # +vote-for-leader 3c7addf4c91bdcffb9d4c4eeea31e0037cac40bc 2\r\n27779:X 10 Jun 11:42:24.264 # slave_host:26379 voted for 806a125c3cb76dd4ee04d97b4a4b8c10377d7d57 2\r\n27779:X 10 Jun 11:42:25.209 # -sdown sentinel master_host:26379 master_host 26379 @ mymaster slave_host 6379\r\n27779:X 10 Jun 11:42:34.877 # -failover-abort-not-elected master mymaster slave_host 6379\r\n27779:X 10 Jun 11:42:34.977 # Next failover delay: I will not start a failover before Fri Jun 10 11:48:24 2016\r\n"}}
{"actor":{"display_login":"mayuraviraj"}}